jshrake/grimoire-legacy

A cross-platform live-coding tool for creative coding with GLSL shaders

grimoire  

What?

grimoire is a cross-platform (Windows, MacOS, "Linux") live-coding tool for creating GLSL shader demos in the style of shadertoy and vertexshaderart. Users write a TOML configuration file that defines resources (image, video, audio, webcam, 3D texture, kinect data) and render passes (vertex shader, fragment shader, primitive type and count, blend state, depth state, uniform samplers). Your shaders, resources, and config file are watched for changes and are live updated at runtime. See the examples below to get started or read the SPEC.md for a detailed description of the configuration schema and runtime behavior.

grimoire is my personal prototyping tool and is in the early stages of development. You may encounter bugs and features may change without notice. Do not expect support. With that out of the way, I think you will find grimoire an easy to use, robust, and powerful tool for prototyping shader effects. Feedback is welcome!

examples: shadertoy compatibility

The following shaders demonstrate compatibility with various shadertoy features. All content is copyright by the original author and licensed under the terms specified, or by the default shadertoy license. If you do not want your work included in this list, you can contact me and I will remove it immediately.

• new, source: cargo run -- ./examples/shadertoy-new
• debug, source: cargo run -- ./examples/shadertoy-debug
• mouse, source: cargo run -- ./examples/shadertoy-mouse
• keyboard, source: cargo run -- ./examples/shadertoy-keyboard-debug
• time, source: cargo run -- ./examples/shadertoy-time
• fps, source: cargo run -- ./examples/shadertoy-fps
• microphone, source: cargo run -- ./examples/shadertoy-microphone
• sound, source: cargo run -- ./examples/shadertoy-sound
• multipass w/ feedback, source: cargo run -- ./examples/shadertoy-deformation-feedback
• video: cargo run -- ./examples/video
• webcam: cargo run -- ./examples/webcam

Install

grimoire tracks the latest version of Rust 2018 stable, which as of this update is 1.34. Note that I actively develop on both Windows 10 and MacOS 10.14 Mojave, so you should have the best experience on those platforms. The project has travis.ci builders for Ubuntu 18.04, but I do not actively test or develop on this platform. If you encounter bugs, please file an issue.

You need to build and install grimoire from source using rust and install the required system dependencies:

• SDL2 for window and input handling
• GStreamer for video, webcam, audio, microphone, and kinect2 inputs
• OpenGL 3.3+, but uses a subset of OpenGL accessible from GLES 3.0

MacOS

$ curl https://sh.rustup.rs -sSf | sh
$ brew install sdl2 gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav

If running on MacOS 10.14 (Mojave), be sure to manually copy Info.plist to target/debug or target/release before running a demo that uses a webcam or microphone resource. The presence of this file allows MacOS to prompt for permission to access the camera and microphone.

If you encounter a build error similar to "Package libffi was not found in the pkg-config search path" you'll need to set your PKG_CONFIG_PATH as described by brew:

$ export PKG_CONFIG_PATH=/usr/local/opt/libffi/lib/pkgconfig

Windows

• Download and run msys2
• Use the x86_64-pc-windows-gnu toolchain: rustup default stable-x86_64-pc-windows-gnu
• Install the required dependencies /w pacman

$ pacman -S mingw-w64-x86_64-pkg-config mingw-w64-x86_64-SDL2 mingw-w64-x86_64-GStreamer mingw-w64-x86_64-gst-plugins-base mingw-w64-x86_64-gst-plugins-good mingw-w64-x86_64-gst-plugins-bad mingw-w64-x86_64-gst-plugins-ugly mingw-w64-x86_64-gst-libav

• Manually copy the SDL2.dll to the top-level grimoire source directory (the one containing Cargo.toml) before running

Note that you need to ensure that your PATH contains the mingw64/bin directory, and that your PKG_CONFIG_PATH lists the directory containing all the .pc files. Since I installed msys2 with scoop, my .bash_profile contains the following lines:

PATH="$PATH:/c/Users/jshrake/scoop/apps/msys2/current/mingw64/bin"
PKG_CONFIG_PATH="/c/Users/jshrake/scoop/apps/msys2/current/mingw64/lib/pkgconfig"

Breadcrumbs:

• https://github.com/sdroege/GStreamer-rs#windows

Linux

$ curl https://sh.rustup.rs -sSf | sh
$ apt-get install libsdl2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav

Build

Rust uses cargo to manage package dependencies and build:

$ cargo build

or with optimizations:

$ cargo build --release

Run

Display help:

cargo run -- --help

To run the default new shadertoy shader:

RUST_LOG=info cargo run -- ./examples/shadertoy-new/

grimoire will watch for saved changes to any file referenced by ./examples/shadertoy-new/grim.toml, including shaders and assets.

See the log crate documentation for more logging levels.

Playback control

• F1: Toggles play/pause
• F2: Pauses and steps back one frame
• F3: Pauses and steps forward one frame
• F4: Restarts playback at frame 0 (iTime = 0)
• ESC: Exit the application

If you are using the keyboard resouce, be sure to avoid these keys. Additionally, you may want to avoid making use of any of the function keys, as I may use these for other features in the future. Note that while toggling play/pause and restarting playback (F1 and F4) work as expected with audio/video resources, F2 and F3 (frame stepping) do not.

Resources

fragment shaders

• Ray Marching and Signed Distance Functions
• Modeling with Signed Distance Functions
• Ray Marching Distance Fields
• The Book of Shaders

vertex shaders

• vertexshaderart lessons
• perspective projection matrix
• full screen triangle

Inspiration

• shadertoy
• vertexshaderart
• bonzomatic
• interactiveshaderformat
• the book of shaders
• https://shadertoyunofficial.wordpress.com/

License

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

grimoire-legacy's Issues

Multi-pass shaders w/ feedback break on window resize

Reproduce by running https://github.com/jshrake/grimoire-examples/blob/master/multi-pass-feedback.glsl and resizing the window. You can resolve the issue by restarting shader playback (press F4) after resizing.

Consider writing a Vulkan backend

Use https://github.com/MaikKlein/ash

consider reading JSON from shadertoy shader 'download'.

i think the ability to 'download all shaders' is something new at shadertoy. it gives you a JSON file of all your shaders. it would be neat if a single entry in those could be read. hm, actually perhaps that's less of a feature for grimoire itself and more like a companion tool.

Consider removing the failure crate dependency

See rust-lang-deprecated/failure#287 for a discussion on the future of the failure crate.

--fps option has no effect

not sure where the bug got introduced,
but in version 0.2.1, hash 7a91982,
on macOS, the --fps option seems to have no effect.
at least, low values such as --fps 2 are rendering way faster than 2 fps.

Update README with fresh examples

• Existing example gifs link to outdated content
• Add some cool multi-pass examples
• Add more vertex shader examples
• Add an example using the kinect
• Add a few of my instagram posts as examples
• Add gifs for all of these that link to the code

Investigate using gstreamer-gl

See https://github.com/sdroege/gstreamer-rs/blob/master/examples/src/bin/glupload.rs

Add more multi-pass examples

Like https://www.instagram.com/p/BvId219hZLP/?utm_source=ig_web_button_share_sheet

Keyboard input doesn't match shadertoy's keyboard input

Compare https://www.shadertoy.com/view/4dGyDm to https://github.com/jshrake/grimoire-examples/blob/master/shadertoy-keyboard-debug.glsl

Consider making gstreamer an optional dependency

Users would no longer be required to install the gstreamer dependencies if they do not intend to use video, audio, microphone, webcam, or pipeline resources.

Add support for Weighted, Blended Order-Independent Transparency

Resources:

• https://en.wikipedia.org/wiki/Order-independent_transparency
• http://casual-effects.blogspot.com/2014/03/weighted-blended-order-independent.html
• http://jcgt.org/published/0002/02/09/paper.pdf
• https://cesium.com/blog/2014/03/14/weighted-blended-order-independent-transparency/

If my understanding of the technique is correct, the only features we need to implement in order to support this feature are:

• Allow clearing individual color attachments
• Allow users to configure the blend state for each color attachment
• Add a flag to disable depth writes (glDepthMask?)

consider adding [an option to] print a message at the start of each recompile.

when saving changes to source,
it's sometimes nice to see positive confirmation that compilation succeeded.

Build and run on the NVIDIA Jetson Nano

Just got one of these https://developer.nvidia.com/embedded/buy/jetson-nano-devkit in the mail! weeeee!

Support image recording

Consider adding a --record command-line flag that captures each frame as an image (png?) to a timestamped directory.

Investigate release binary size

Is it possible to get below 1 MB? 64k? 4k? Without gstreamer, file watching? No std? A man can dream!

Support specifying preprocessor #defines in the pass configuration

See https://github.com/jshrake/grimoire/issues/61#issuecomment-494402957

Make SRGB framebuffer configurable

I keep manually commenting and uncommenting glEnable(GL_FRAMEBUFFER_SRGB) to match various source material. Make it a configurable option, either at the command-line or in grim.toml. Of note, I believe at one point I disabled it to match source material on shadertoy, and then enabled it to match source imagery from http://jcgt.org/published/0002/02/09/paper.pdf while working on https://github.com/jshrake/grimoire/issues/34. I'm once again disabling it while working on #21

Update gleam to 0.6 and regex to 1.0

As suggested in #9

Add kinect example

Port https://github.com/jshrake/grimoire-examples/blob/master/kinect2-depth.glsl and make sure it still works

Consider renaming the "audio" resource to "sound"

I type sound every time and it never works. I think it's a sign.

shadertoy-microphone example looks wrong

Support clearing only the depth buffer

Currently, we can clear the color buffer, or the color buffer and the depth buffer, but not just the depth buffer. Make it so!

Support drawing a pass multiple times

We should be able to draw a pass multiple times in a row (w/ feedback) by specifying something like loop = N in the config.

video example no longer runs

thread 'main' panicked at 'Could not canonicalize file name: Os { code: 123, kind: Other, message: "The filename, directory name, or volume label syntax is incorrect." }', src\libcore\result.rs:1165:5

Relative filesystem paths for audio & video resources don't resolve on windows

Specifying a relative filesystem path for audio & video resources is broken on windows

The webcam resource only works on MacOS

autovideosrc is not an element on windows. The webcam on window uses the ksvideosrc element.

Document using the F{1,2,3,4} keys for playing, pausing, stepping, and restarting playback

This already works, just needs documenting in the appropriate place!

F1: Toggles play/pause
F2: Steps back one frame
F3: Steps forward one frame
F4: Restarts playback (iTime -> 0)

Note that toggling play/pause and restarting playback (F1 and F4) work correctly with audio/video resources. However, F2 and F3 (frame stepping) do not work as expected with audio/video resources.

Consider adding support for geometry shaders

My immediate use case is to generate surface normals in the geometry shader when doing procedural modeling in the vertex shader. ATM, I'm generating the surface normals in the vertex shader which is awkward and likely bad for performance as the triangle count scales.

Users should be able to configure the framebuffer depth attachment

• Users should be able to disable the depth attachment
• Users should be able to sample from the depth attachment

Add support for compute shaders

Add support for instanced rendering OBJ models

Users should be able to specify the framebuffer size with a scale parameter

Currently users can specify the width and height in pixels of a framebuffer resource. It would be nice if they could instead specify a scale parameter relative to the window size.

add command-line option to position window

eg "gravity" or something simpler like Position.

this would help me a lot when quitting and restarting Grimoire frequently.

FBO color attachment swapping should only occur after a feedback render pass

need to think through this some more

Consider migrating from SDL2 to winit

winit is a window handling library in pure Rust. Ideally grimoire would only depend on pure rust crates. Note that ggez is making a similar migration ggez/ggez#391

I would like to understand how this would impact the ability to get grimoire running on mobile (iOS, Android), web (emscripten), and various VR/AR platforms.

Document latest features

I've landed several new features that need documentation:

• restart-on-save
• pass loop (draw a single pass multiple times per frame -- really useful for reaction-diffusion sims!)
• geometry shaders
• separable blend states
• --record

Add uniform for mouse scroll wheel

Consider migrating from gleam to the gl-rs crate

gleam is intended for internal use by servo. Instead, consider depending on https://github.com/brendanzab/gl-rs

Make interface for live updating uniform values

I would like to have a IMGUI-like interface for updating generative parameters (uniform values). This interface does not have to be rendered in the same application and instead could be a web page that communicates with the application over [insert technology here].

Users should be able to specify the framebuffer pixel format

Currently, the buffer format is hardcoded to RGBA. User should be able to specify 1, 2, 3, or 4 components. RGBA (4 components) by default is fine.

Audio resources don't match Shadertoy

Running the microphone, audio, or tuner examples side-by-side with the shadertoy equivalents produces slightly different results. This is likely due to my own misunderstanding of audio and inexperience with the gstreamer API. The high-level goal is to produce the same values from the AnalyserNode Web API with gstreamer and the spectrum plugin.

A few notes:

• Shadertoy seems to be using the default properties of the AnalyserNode (-100 max dB, -30 min dB, 2048 for the fftSize (and the frequencyBinCount is half that, so 1024. However the resulting texture described by shadertoy documentation has only 512 width! confusion!)
• The spectrum plugin only produces the frequency data. The wave data has to be acquired through a different code path. (See audio.rs for more details).
• The spectrum plugin does not do any smoothing.

Consider writing a rendy or wgpu-rs backend

https://gfx-rs.github.io/2019/03/06/wgpu.html

add boolean preprocessor symbol GRIMOIRE or similar

currently shaders can be copy-and-pasted from the grimoire environment right into ShaderToy (yay!) with the minor modification of removing the statements defining fragColor and main() which appear at the bottom. eg the 'new' example.

however, if Grimoire injected something like #define GRIMOIRE before sending the shader code off for compilation, then that minor step of removing those lines could be eliminated.
ie the code would become

#ifdef GRIMOIRE
out vec4 fragColor;
void main() { mainImage(fragColor, gl_FragCoord.xy); }
#endif 

i'm note sure what the actual best symbol would be, eg if there's already precedent for something like ENVIRONMENT_GRIMOIRE etc, nor the best way of injecting such a symbol. if i were doing it, i'd probably just literally prepend the line #define GRIMOIRE to the front of each shader file, but that seems hackish.

an alternative approach might be to implement something similar to Shadertoy's common tab, but only functional in grimoire. eg grim_common.glsl. then one could add whatever #defines one pleased in there. however with a view to establishing a consistent way stuff is done, i'd probably opt for the other approach first.