efairbanks/cybin

A programming/performance environment for seamless online and offline audiovisual rendering.

Cybin

v (thumbnail links to intro video) v

Cybin is a dependency-light, LuaJIT-based audio programming environment suitable for realtime performance and offline rendering. Unlike many popular multimedia programming environments, it strives to do everything in one place all at once, and its strength and weakness lies in completely reinventing the wheel in order to offer the user sample-accurate timing and complete control.

The best way to understand Cybin is to understand what it does, and by extension, what it does not do.

What it does is provide the user with:

• a Lua interpreter that calls a yet-to-be-defined function called __process that recieves an arbitrary number of audio samples (one per channel) and outputs an arbitrary number of audio samples (one per channel) every 1/cybin.samplerate seconds
• a global table called cybin that provides some basic audio/music utilties that would be difficult or impossible to provide adequately using a Lua interpreter alone (audio system info, MIDI I/O, audio file handling, etc...)
• a set of command line arguments that may be used to configure your Cybin environment
• multiple optional libraries that contain opinionated sets of tools for creating music and manipulating audio in Lua
• a seamless way to transition from realtime performance and production to offline rendering

What it does not do is:

• tell the user how to make music
• tell the user what is or is not music
• tell the user it should be used to make music
• tell the user it should or should not be used for any particular purpose
• anything else

Installation:

• Make sure you've got a C++ compiler
• Install LuaJIT, JACK, and libsndfile (MacOS dependencies are available through brew)
• Install git
• Run cd && git clone https://github.com/efairbanks/cybin.git && cd cybin
• Run ./LINUX_BUILD.SH or MACOS_BUILD.SH depending on your platform (Windows support hopefully coming soon)
• Put export PATH=$PATH:~/cybin in your .bash_profile or other shell startup file (or symlink to the cybin executable, whatever floats your boat)

Optional

• Install Emacs
• Place the following into your Emacs init file (~/.emacs, ~/.emacs.el, or ~/.emacs.d/init.el)

(add-to-list 'load-path "~/cybin")
(autoload 'cybin-mode "cybin-mode" "Cybin editing mode." t)
(add-to-list 'auto-mode-alist '("\\.cybin$" . cybin-mode))
(add-to-list 'interpreter-mode-alist '("cybin" . cybin-mode))

Emacs Hotkeys

• C-c C-c -> execute line
• C-c C-b -> execute file
• C-c C-e -> execute block/paragraph

caveats: The way emacs traditionally sends blocks of code to subprocesses has something like a 1kB character limit, so large blocks should be broken up or imported using dofile() or require().

Goals:

• Few dependencies (relatively speaking) (done)
• Real-time, from-scratch audio synthesis in LUA (done)
• Real-time, from-scratch video & image synthesis with LUA and OpenGL (tbd) (this coming eventually as a separate-but-related utility)
• Seamless transition from live performance to high-quality offline rendering (done)
• A robust standard library of real-time-capable audio manipulation and synthesis utilities all written in Lua (done)

Dependencies

• LuaJIT
• libsndfile
• JACK

cybin's Issues

Build in some transparent-but-optional bootstrapping

Working with Cybin for the first time should be as simple as running a one-liner that defines a musical pattern, but users should also be empowered to design their own music system entirely from scratch. How do we achieve this without obfuscating the inner-workings of Cybin? Perhaps a super-powerful, modular, customizable synth like SuperDirt built into Cybin/Lua with its own scheduler that can receive musical data structures and automatically play them?

Yeah, that sounds about right.

OpenGL/GLSL/GLOO -> LuaJIT

Create LuaJIT bindings for (re)loading OpenGL vertex and fragment shaders and for supplying those shaders with uniforms.

It would also be a good idea to perhaps think about Lua code that can intelligently generate and organize GLSL code. This would allow for features like:

• a Lua DSL for procedural graphics
• a tiled windowing system within an OpenGL context
• sound shaders?????

Performance profiling and optimization

Right now LuaJIT is getting excellent performance, but this can be further optimized. Profiling should be done to determine what kind of savings could be gained by modifying buffers in-place, handing off synthesis/DSP work to C++ when fine control from Lua is not required, etc...

Comparable implementations of BlackBox classes should be created in Lua and C++ and their performance should be compared.

Update Emacs keybindings

The way cybin-mode defines keybindings makes it kind of difficult to customise since it uses this prefix thing:

cybin/cybin-mode.el

Line 300 in 91a797b

(defvar cybin-mode-map

But anyways, it would be good to have:

(define-key result-map (kbd "<C-return>") 'cybin-send-paragraph)
(define-key result-map (kdb "C-c C-q") . cybin-kill-process)

Fix examples and figure out a way to get cylibs reliably in path

some of the examples are screwy and the libs don't load or they don't run as expected in either atom or emacs

we need this to be consistent so we're not alienating new users

REPL for Cybin

Cybin should have some kind of REPL with integration into a popular text editor (or two) like Atom/vim.

Example `euclidbreak.cybin` is broken?

Can't get dofile('euclidbreak.cybin') to work...

Seems that the imports are different to the other files?

dofile('utils.cybin').load()
dofile('pattern.cybin').load()
dofile('melody.cybin')
dofile('ugen.cybin')
C=dofile('clodule.cybin')
mod=math.fmod

The others use require. But even when I try to change above to require I get a stack overflow

Restructure repo dirs

Everything is currently inside examples

BootCybin file

E.g.

os.execute('jack_connect cybin:audio-out_1 system:playback_1')
os.execute('jack_connect cybin:audio-out_2 system:playback_2')
dofile('/home/cybin/examples/cylibs/clodule.cybin')
dofile('/home/cybin/examples/cylibs/ugen.cybin')

Add support for parsing command line flags

Add in support for parsing command line flags and loading configuration details via those flags.

Command line flags will be used for things such as:

• online vs. offline rendering
• time offsets
• loading additional Lua files & libraries
• setting configuration elements such as sample rate or resolution

Implement all BlackBox lib in Lua

This includes oscillators, filters, envelopes, granular synths, polyphony, etc...

This should be implemented with a sensible inheritance model and with factories where appropriate. For instance, you should be able to create a new subclass of type Voice by supplying it with a custom oscillator and envelope, and then create a custom subclass of Synth by supplying it with your custom Voice class.

Support remote control via OSC

Integrate something like LibAV for import/export of audio/video/media

Cybin should be able to import and export various kinds of multimedia content. We're gonna need an external library for that. Probably what ffmpeg uses.

LibSoundIO sometimes fails to start in MacOS

It seems that sometimes MacOS will complain about invalid memory usage when soundio_default_output_device_index() is called during initialization. It's unclear what causes this, but it is not an issue in Linux.

Update README for macOS + Emacs, re Ruby + Full Disk Access

https://apple.stackexchange.com/questions/371888/restore-access-to-file-system-for-emacs-on-macos-catalina

an ordered series of simple, digestible tutorials with inline comments

the current examples are extremely daunting and not very helpful

this needs to be fixed pronto

Cylibs and Examples inline documentation

LDoc approach: https://github.com/stevedonovan/LDoc / http://stevedonovan.github.com/ldoc/

Doxygen approach: https://github.com/alecchen/doxygen-lua/blob/master/example/example.lua

Lua DSL for SDF ray-marching

This would just be cool. Could just be a library. IDK.

Update documentation around boot.cybin file and new plugins

Build musical patterning library

Build a musical patterning library that fuses techniques drawn from:

• TidalCycles
• My Lua Seq custom table implementation from my Renoise scripts
• My C# BlackBox code
• My Overtone/SuperCollider compositions

Should patterns/sequences be aware of time? I want time to be in cycles, but what does that mean for patterns/sequences? Should they be query-able? Should they be transformed into queryable event streams? What is the difference between what the event scheduler receives and operates on vs these?

Add optional file-watching capabilities

Cybin should be able to reload Lua scripts in real time periodically if asked to do so.

Get Cybin compiling in Windows

I haven't tried compiling Cybin in Windows. Theoretically, it should...

I really don't love compiling in Windows.

Allow configuration of Cybin environment via LUA script

Allow Cybin users to configure the Cybin environment through variables loaded in via a LUA script.

Examples:

• sample rate
• resolution
• interpolation
• supersampling
• audio/video device
• channels

Multiple options for handing off audio data to C++ audio system

Right now Cybin calls a Lua function called __process(sr) that accepts the current sample rate and returns a float representing a single audio sample. Not only should this be modified to support multichannel audio, there should be a way to configure how audio is passed from Lua to the underlying audio system and what that system's responsibilities are.

For instance, perhaps we would rather work with an audio buffer of size N that provides frequency domain and time domain representations instead of a stream of samples.

Design and implement an event scheduling system

This scheduling system should be loosely based on Tidal's because Tidal's scheduling system rules. It will also probably take some design cues from my earlier work with Overtone. This system needs to be simple enough that people want to use it but flexible enough to not be prescriptive. (manual temporal recursion is not intuitive)