artiphishle/daw

Audio DAW

Digital Audio Workstation

---

Kauderwälsch

There are some terms/abbreviations used in the code, here the list:

Const	Meaning	Description	Reason
$	css	Component styles	reserved, DI* planned
$d	data	Component data	reserved, DI* planned
$e	event	Component event	reserved, elsewhere placed should throw

* Dependency Injection

🎼Quickstart

Assuming you have Node.js installed...

npm i && npm run dev

🥁 localhost:3000

---

Table of contents

• 🎼Platform strategy

• 🎼Instruments

• 🎼PFUI design

---

Vision

Free & open-source

This platform will be free to use for everyone, and it will be open-source. Of course we'd like to offer you additional services that could come with a price tag, but the core of the platform will always be free.

Compose, arrange, record, mix & master

Yeah. It's a DAW, right?

Intuitive & AI supported workflow

The platform will help you while you're working on your project. E.g. you can turn on rules that will warn you if you are out of tune, or transpose your music to a different key. You can also tune the instruments to Verdi's A4 (432Hz) with one simple click if you like to.

There will be tons of free samples, grouped by tons of genres, there will be live workshops while you're creating your next hit. You can also collaborate with other musicians, share events and much more.

Vision: eLearning

Audio-Engineer

You will learn how to master the art of mixing, there will be many chapters with different topics, like: Harmonic EQ'ing, Psycho Acoustic, New York Compression, Cambridge HPF, etc.

Did you know? The SM57 & SM58 live microphones mainly used for vocals on live concerts sound great to record an electical guitar! The SM57 will give you a more direct sound, the SM58 sounds more airy instead.

Learn to play an instrument

You will learn how to play an instrument, the better you become the more influence in the online community you will learn build & maintain. You will learn about existing playing styles, and there will be exercises like online games to play with other musicians.

(Game-) Sound Design

You will learn how to compose sound to movies/game and how to create sound from scratch. This includes Foley and orchestration, how to design emotion for the listener and plan your project like a pro.

Did you know? If you press E & F on your piano in all octaves fast and furious, you can see Tom haunting Jerry. If you play them upwards (EF EF EF EF) it will increase the tension, vice verca Tom has already resigned catching that Meow!

🎼Contribution

/**
 * You like coding?
 * You are a musician</>
 * You like music
 */
export default function Collabo() {
  return EMotion.Perfect;
}

We're @here 🙋‍♂️ and @there

---

🎼Screenshots

Design will be applied soon, this is just a raw sketch!

---

🎼Changelog

🍂 October 2023

October, 25th

Nice Gold Knob
(Design by ykadosh)

November, 17th

• Auth Account create/edit/delete/login/logout
• OAuth2 Google/Facebook/Apple/Mail/Github/Notion/Phone/etc.

2024

January, 22th

• License MIT license added, feel free to add some credits to 'artiphishle' Github account or Instagram account with the same name. It's also very appreciated if you decide to become a collaborator in the main repo instead of forking (unless there is a good reason obviously). Thanks!

October, 24th

• Arranger Track selection (active track)
• Arranger Toggle note events

October, 22th

• Mixer Route tracks to channels

October, 19th

• Theme Start to setup customized theming & added PrimeReact
• Pfui Menu/ContextMenu one4all menu. Bind a Menu/Contextmenu to a target area

October, 14th

• Mixer FFT Analyzers for all tracks
• Cypress Component & E2E tests (removed Jest)

October, 13th

• Progression Show famous chord progressions by tonic (try to change clef on top)
• E2E Testing using Cypress, also integrated into CI pipeline & first test for DnD useSortable tracks (Arranger)

October, 12th

• Arranger Players & Sampler can play Tone.Sequence
• Samples added new sample library

October, 7th

Parts of Sequences playable in Players and Sampler Free Samples updated to the public domain

October, 5th

• Github Workflow lint, test & build on 'main' push & PR (matrix strategy)

October, 4th

• Arranger Quantization grid & playing sequences
• SWR 'Selector' to handle unserializable data

🍃 September 2023

September, 29th

• Portals The instruments are loaded at the same location via usePortal hook
• Refactoring of the whole project (more soon, project 2 months young)
• Navbar Redesign redesign & new: update measureCount & quantization
• Sampler playable open the instrument via the 'routing' input field in the mixer
• Arranger & Mixer auto-resize throttled update on window resize
• Browser Media browser tab, ready to be filled with provided free samples

September, 23th

• BPM Detection works, audio tracks are automatically analyzed
• MIDI Device support for Chrome, more to follow (Tab 'Settings')

September, 20th

• Add/remove notes in MIDI tracks

September, 15th

• Mixer Volume meter, ready to add inserts/sends
• SWR Instruments are now loaded with ProjectContext

September, 13th

• Store project settings using SWR
• Arranger playback (drums)

☀️ August 2023

August, 24th

• Jest Setup using Rust compiler
• Locator & Transport playable sound
• Basic Pitch AI to analyse audio files to generate:
• Generate song sheets using ABC.js

August, 22th

• Arranger DnD tracks
• Locator visualize transport position
• MultiLanguage 🇨🇭 🇩🇪 🏴󠁧󠁢󠁥󠁮󠁧󠁿 🇫🇷 🇮🇹

August, 9th

• Basic Synths & FX using Tone.js
• Transport (playback, bpm).
• Chord Progression generated by tonic & roman letter input
• AudioKeys use your keyboard as a MIDI device
• Verdi's A a hook to tune ♩A4 of instruments to 432hz
• PFUI Pretty Futuristic UI

daw's Issues

StoryBook Setup

Add Storybook as a and move pfui variations of components into it

Acceptance Criteria

• Storybook runs properly
• Most of UI components are migrated
• Install suitable plugins

Volume meter

Display the played volume per channel (also in the stereo out)

Acceptance Criteria

• Reusable Volume component (ATM <Db /> in the mixer.tsx)
• Use: Show current Volume for all channels
• Update the float number, e.g. Volume is: -2.54 accordingly (2 decimals)

PianoRoll

Let's create a PianoRoll to record MIDI pianos later.

Acceptance Criteria

• Reusable <PianoRoll /> component

Authentication

Provide User Authentication

Acceptance Criteria

• Register account
• Edit profile
• Delete profile
• Login
  • Github
  • eMail
  • Phone
  • Facebook
  • Apple
  • Notion
  • etc.

CI Pipeline (Github Actions)

Create a CI pipeline that includes linting and testing.
It's possible that there is no (Jest) test file yet, create a simple one.

Acceptance criteria

• PR's & Pushes are tested:
  • npm run test must pass
  • npm run lint must pass

Sortable Tracks only on the left

ATM The midi arrangement cannot be changed because the DnD Event is everywhere on the track

Acceptance Criteria

• Sortable tracks with a "Draggable" Icon on the left
• Verify midi arranger can be used

Audio to MIDI

We'd like to import any .wav audio file and detect instruments played and generate midi tracks from it.

Input:

• Github: basic-pitch-ts

Acceptance Criteria

• Add a new MIDI plugin that allows DnD (use dnd-kit, already installed) a .wav file into it
• .wav files dropped into MIDI track that uses the plugin will be converted automatically to MIDI tracks inside of the parent
• Use the drum MIDI map (see drum plugin in midi folder)

So, to be clear, the expected result when dropping a .wav file with only BaseDrum and Closed Hihat

[ MIDI TRACK can be expanded ]
-> [ Audio .wav file dropped here ]
-> [ Ch. 35 Basedrum ]
-> [ Ch. 42 Closed Hihat ]

BPM Detection

We need to automatically detect BPM of audio files.

Acceptance Criteria

• BPM Detection for Audio Tracks

Audio Track Length

Let's draw the AudioTrack files in Tone Player AND WaveSurfer to get insights.

Acceptance Criteria

• Update on windows resize
• Update on measures change
• Update on quantization change

Arranger Group Nesting

Group channels should be the parent of their members.

Acceptance Criteria

• Use dndKit to make tracks nestable
• Update (mutate) the projectSettings & routing accordingly

[PFUI] ContextMenu

Create a ContextMenu UI to reuse (also as a menu if normal click)

Acceptance Criteria

• Use recursion for multi-level menus
• Any left click inside or outside of the menu makes it disappear
• Bind actions to menu entries (this might lead to architectural thinking on how to trigger actions in general)

[Reusability] Quick Enhancement

Time to sort things out. The creation of instruments until destination is spread over different files, and belongs into a separate module 'Instrument', we can also move global project settings into a <Project /> component.

Acceptance Criteria

• Create<Project /> & <Instrument />
• Instruments use same useAudioLoader hook
• Handle all instruments together instead of the if and elses

Transport position

Create a new view to display the current location of Transport.

Acceptance Criteria

• New Position.tsx similar to components/tracks/TimeTrack.tsx
• Display the current location of Transport using: Transport position

UI Grid

Create a new UI Grid, we will use this for Drum Sequencer, etc. later.

Acceptance Criteria

• UI Grid with custom rows & cols
• Support SubGrid (1 level ok)
• Actions: onClick

Setup Unit Tests (Jest)

Let's make sure that the Unit testing framework is ready to use.
As Jest is already integrated, let's use it, I'd suggest to also use Cypress, but for now some unit test would be great.

Accpetance Criteria

• Setup Jest
• Rust Compiler
• jest.config.mjs
• at least one passing test

[API] Endpoint Separation

Let's clean up the api and use distinctive endpoints.

Acceptance Criteria

• Generic description of the API (JSON) then we can generate API's later (Mongoose, Supabase/Postgres, FastAPI, etc.) ~> follow up task will be useSWR or ReactQuery hook generation

• Don't forget to describe selectors, e.g. a Tone Instrument cannot be stored serverside and has to be handled before mutation and after fetch

API

Project

Method	Endpoint	Interface
GET	/project	IProject
PATCH	/project	Partial
GET	/project/presets	IProjectPreset[]

Track

Method	Endpoint	Interface
GET	/tracks	ITrack[]
PATCH	/tracks	ITrack[]
GET	/track/{id}	ITrack
PATCH	/track	Partial

Public Samples Directory

Method	Endpoint	Interface
GET	/directory	IDirectory

Dsp

Method	Endpoint	Interface
GET	dsp/inserts/{trackId}	IDspInserts
GET	dsp/sends/{trackId}	IDspSends

Ambition

Screen 'daw'

<App>
  <main>
      <Song>
          <Arranger>
              <Tracks>
                 <TimeTrack />
                 {/* Example Track Config  */}
                 <AudioTrack />
                 <MidiTrack />
                 <ChannelTrack />
                  {/* Master Channel */}
                  <ChannelTrack />
              </Tracks>
          </Arranger>
          <Tabs>
            <Mixer />
            <PianoRoll />
            <Browser />
          </Tabs>
      </Song>
  </main>

  {/* Open Instrument */}
  <InstrumentPortal>
      <Instrument />
  </InstrumentPortal>
  {/* Send Effects */}
  <FxRackPortal>
    {/* Example Rack Config */}
    <FxRackComp />
    <FxRackDelay />
    <FxRackVerb />
  </FxRackPortal>
</App>

Screen 'edu'

<App>
    <main>

      {/* Example Course */}
      <Course>
        {/* todo */}
      </Course>
    </main>
</App>

Cypress API mock

Let's fix the e2e test, assuming due to API requests (it works locally) to git lfs which isn't installed in CI pipeline

Acceptance Criteria

• Choose how to quickly fix the test
• Avoid API requests
• Create stubs for Tone.js or mock it all

[Bug] Fix Transport Stop

Transport is not stopping when clicking on ⏹️

Acceptance Criteria

• Transport stops as expected

UI Tabs

Tabs aren't interactive yet, we want to switch tabs when clicking on the tabNav.

Acceptance Criteria

• Finish the UI tabs component
• Remove the existing one (just keep the Mixer)
• Put the Arranger & Sheet into new Tabs for now

useAudioRecorder

We want to record vocals, using the user's local microphone.

Acceptance Criteria

• Stream live to Tone.Player
• Don't upload the audio file (user home directory required in other task)

[Update] Tabs & Splitter

Bring back the existing features (from Tabs) of before.
Also add a PrimeReact Splitter to make the top/bottom separation resizable.

Acceptance Criteria

• Tabs
  • Sheets
  • Tests
  • Browser
  • SignalFlow
  • PianoRoll
• Splitter (resizable)

RxJS or SWR Integration

The project settings should be managed in a non-chaotic way, some components share properties, some are nested, some will get data from a db or from anywhere, while other data comes from local configs, so it's time.

Input:

• RxJs with React Hooks

Acceptance Criteria

• Let's subscribe to the settings we want to
• Think about an appropriate way and implement it
• Use hooks and keep thinks flexible

[Chord] Piano Progression

Let's play a PIano Progression (chords) using a Sampler.

Acceptance Criteria

• Use existing Piano Sampler
• Generate chord progressions using current clef

[PFUI] Knob

Let's use the Knob from <Adsr /> and make it available in PFUI

Acceptance Criteria

• New UI element <Knob />

Project: Load & Create

don't load projectSettings automatically, show a Welcome screen to create/load a project instead.

Acceptance Criteria

• Load 1-2 genre presets do choose from, and the (existing) default one
• Don't load projectSettings, wait until user chooses

Synthesizer GUI

We want a synth that combines all the Tone.js Synths.

Acceptance Criteria

• Create <OmniSynth />
• Start with one Osc and provide all types
• Add Output Gain (PFUI Knob)
• Add Polyphony
• Add Chords
• Add MidiTracks with different settings
• Fetch/Mutate settings
• Show in InstrumentPortal

[Midi] Toggle Events

Let's add/remove Midi Events in the Arranger by mouse click

Acceptance Criteria

• Midi events can be added/removed by mouse click (toggle)
• Correct playback of new drawn events

[Mixer] Fix 'Meter'

In the mixer, the louder tracks have a quieter meter, -5 is louder than -17 as an example.
The master should be the sum of all the tracks, this wasn't done yet either.

Acceptance Criteria

• All the mixer tracks show correct meter's

Music Sheet

Let's generate a music sheet from the arranged project.

Input:

• abcjs

Acceptance Criteria

• Convert notes to Markdown
• Render Sheet

[Cypress] Setup

To provide reliable e2e testing let's setup Cypress and add a meaningful test.

Acceptance Criteria

• Setup Cypress
• Arranger: Add a test for sortable tracks
• Add scripts to package.json
• Add to ci script and GitHub Workflows

AudioTrack with Waveform

Load .wav file (no import, just load) and show its waveform. Play it as well.

Useful:

• Waveform

Acceptance Criteria

• Load .wav in <AudioTrack />
• Play the .wav file
• Visualize the Waveform

[3D] Animated Drums

Let's do some 3d modelling and start with the drums!

Impact:

1. Find a model (watch polygon count, not too many)
   • possible formats: .c4d, .fbx, .obj, .glb, .gltf
   • platforms:
     • cgtrader.com (paid)
     • 3dwarehouse (free)

Acceptance Criteria

• Model in a format described above
• API for at least BaseDrum & Snare (more if easy)

[FX] Frequency Analyzer

Let's build a frequency analyzer to visualize what's sound!

Acceptance Criteria

• usage of <Canvas>
• adaptable amount of frequencies

Fix 'instrument is a string'

After toggling a MIDI note in the arranger, the next one in the same track cannot be played.

Acceptance Criteria

• toggling in any MIDI track is no issue, and remaining notes are played as expected

Instrument: Sampler

Let's create a sampler, and add some free sample libraries as well (Git LFS)

Acceptance Criteria

• Provide some free sample libraries (nice ones)
• Create a Sampler & load/play some sounds in a new track

Sortable Tracks

Finish the functionality "Sortable Tracks" with the already installed library DnD Kit:
https://docs.dndkit.com/presets/sortable

Acceptance Criteria

• Tracks are sortable (Config is updated onDrop)
• Synths are draggable

[DSP] Visualization

Acceptance Criteria

• Install ReactFlow
• Draw the described signal flow

😼After this task we're able to automatically apply the signal chain by drawing it, and generate visualization based on signal flow configs

Change Clef, MC, Q

There is no way to change measureCount, quantization & clef.

Acceptance Criteria

• Add the required entities to Transport
• Mutate onChange
• Progression: Update by clef
• Arranger: Update by measureCount
• TimeTrack: Update by quantization

Draggable Instruments

Let' s make the instruments draggable, headless of course!
Additionally update the channel sort order in the mixer when useSortable on tracks in arranger.

Input:

• useDraggable

Acceptance Criteria

• Instruments can be dragged to another position
• Mixer tracks update their sort order according to tracks in arranger

[PFUI] Formik Integration

Create a new UI Form element to build forms using Formik as a plugin (generic, not only formik is an option later).
Then add a newsletter subscription form to a new tab (before settings)

Note: This tasks ensures creation of forms, you cannot save the data yet, as we have to add a db first ;)

Acceptance Criteria

• Usage of Formik or explain why not and propose another library
  - Create a reusable NewsletterForm component and add a test to make sure the entered eMail address is valid

Browser – load 'public' folder

Let's show the samples in public/ and prelisten them too

Acceptance Criteria

• Move API from pages/api/ to app/api/
• Three Component
  • useSortable()
  • open/close
  • selectable
• Content side
  • show the content of selected folder from the tree
  • audio files can be prelistened to

[Nx] Setup

We need to extract livraries and assign them to appropriate repos.

Acceptance Criteria

• pfui (UI lib will be reused, main reason)

• colorfuel (this will become great soon)

• fork new instrument libraries (git lfs) as a separate workspace too

i18n Integration

ATM you have to manually change the language in the code which is nice for developers but now it's time to integrate internationalization correctly.

Acceptance Criteria

• Choose the best package to use with Next.js
• Integrate the package correctly
• Provide fallbacks, eg. if de_CH but only de_FR available, fallback to de
• Update <html lang=''> (if not automatically ensured)
• Add a language switcher to the footer (right side), preferrably with a flag in a select box

Arranger – Sequence algorithm

Draw the sequence correctly into the Arranger to make sure it works.
Also find a way to provide the time (in Notation, e.g. 4n) and update the interfaces.
It's important to verify this task using unit tests.

Acceptance Criteria

• TDD: write the unit tests first
• Quarter notes normal & with subdivison
  • ["C3", "C3", "C3", "C3"]
  • ["C3", "E3", ["D4", "E4"], "C2"]
• Eight notes normal & with triplets
  • ["C3", "C3", "C3", "C3", "D3", "D3", "D3", "E3"]
  • ["C3", "E3", ["C4", D4", "E4"], "C2"]
• Data structure:

          events: {
               notes: Note[]
               time?: Tone.Time  // "4n"
          }
      }
  

Track Selection

ATM it's not possible to select a track in the arranger

Acceptance Criteria

• Toggle select track
• Apply a selected style on the selected one
• Save the selected info to the projectSettings

Replace StepSequencer, use MIDI track

Use one MIDI track specific for drums (use Accordion UI to open drum tracks inside of the MIDI track) and make it playable.

Acceptance Criteria

• Drums are playable in a single MIDI track
• Use the already added Accordion UI
• Use the official Drum Map:

MIDI Controller

Let's connect a MIDI controller and play some sound on it!

Input:

• MIDI Controller

async function connectMIDI(){
  const access = await navigator.requestMIDIAccess();
  const devices = access.inputs.values();
  // iterate devices
}

Acceptance Criteria

• Connect a MIDI Controller
• Route the MIDI signal to a synth and play sound

Tailwind Theme

Let's define a basic template for the app, also include darkmode if easy to do so.

Acceptance Criteria

• Add a Tailwind Theme
• Add DarkMode if fast, otherwise add a new task
• Extract all the existing className's and use Theme instead

WAM & WebAssembly

Evaluation of WebAssembly Plugin implementation

Some input:

• Web Audio Modules 2.0
• Web Audio Modules (Europa Synthesizer)

Acceptance criteria

• Research (and add information here to the issue)
• Answer whether it's possible to use 'real' .vst, .vst3, or .au plugins
• Implementation of a simple plugin in the recommended way

Arranger Midi Editor

Update (mutate) changes in Midi tracks to be able to perform.

Acceptance Criteria

• Refactor Track (no redundant JSX)
• All MidiTrack's can be used to perform
• Update (mutate) the projectSettings accordingly