vito/bass

a low fidelity scripting language for project infrastructure

bass

Bass is a low-fidelity Lisp dialect for the glue code driving your project.

README.mp4

reasons you might be interested

• you're sick of YAML and want to write code instead of config and templates
• you'd like to have a uniform stack between dev and CI
• you'd like be able to audit or rebuild published artifacts
• you're nostalgic about Lisp

what the thunk?

Bass is a functional language for scripting commands, represented by thunks. A thunk is a serializable recipe for a command to run, including all of its inputs, and their inputs, and so on. (Why are they called thunks?)

Thunks lazily run their command to produce a stdout stream, an output directory, and an exit status. These results are cached indefinitely, but only when the command succeeds.

$ bass
=> (from (linux/alpine) ($ cat *dir*/README.md))


        ██████
      ██████████
    ██████████████
    ████  ██  ████
    ██████████████
    ██████  ██████
  ████    ██    ████
    ████      ████

<thunk JU61UMJQ70FMI: (.cat)>
=> (thunk? (from (linux/alpine) ($ cat *dir*/README.md)))
true

To run a thunk and raise an error if the command fails, use (run). To get true or false instead, use (succeeds?).

=> (def thunk (from (linux/alpine) ($ echo "Hello, world!")))
thunk
=> (run thunk)
; Hello, world!
null
=> (succeeds? thunk)
true
=> (succeeds? (from (linux/alpine) ($ sh -c "exit 1")))
false

To access a thunk's output directory, use a thunk path. Thunk paths can be passed to other thunks. Filesystem timestamps in thunk paths are normalized to 1985-10-06 08:15 UTC to support reproducible builds.

=> (def thunk (from (linux/alpine) ($ cp *dir*/README.md ./some-file)))
thunk
=> (run (from (linux/alpine) ($ head "-1" thunk/some-file)))
; # bass
null

To parse values from a thunk's stdout or from a thunk path, use (read).

=> (next (read (from (linux/alpine) ($ head "-1" thunk/some-file)) :raw))
"# bass\n"
=> (next (read thunk/some-file :lines))
"# bass"
=> (next (read thunk/some-file :unix-table))
("#" "bass")

To serialize a thunk or thunk path to JSON, use (json) or (emit) it to *stdout*. Pipe a thunk path to bass --export | tar -xf - to extract it, or pipe a thunk to bass --export | docker load to export a thunk to Docker.

$ ./bass/build -i src=./ | bass --export | tar -xf -
$ ls bass.linux-amd64.tgz

This, and generally everything, works best when your thunks are hermetic.

tl;dr

It's a bit of a leap, but I like to think of Bass as a purely functional, lazily evaluated Bash.

Instead of running commands that mutate machine state, Bass has a read-only view of the host machine and passes files around as values in ephemeral, reproducible filesystems addressed by their creator thunk.

example

Running a thunk:

(def thunk
  (from (linux/ubuntu)
    ($ echo "Hello, world!")))

(run thunk)

Passing thunk paths around:

; use git stdlib module
(use (.git (linux/alpine/git)))

; returns a thunk dir containing compiled binaries
(defn go-build [src pkg]
  (subpath
    (from (linux/golang)
      (cd src
        ($ go build -o ./built/ $pkg)))
    ./built/))

(defn main []
  (let [src git:github/vito/booklit/ref/master/
        bins (go-build src "./cmd/...")]
    ; kick the tires
    (run (from (linux/ubuntu)
           ($ bins/booklit --version)))

    (emit bins *stdout*)))

irl examples

• Bass
• Booklit

what's it for?

Bass typically replaces CI .yml files, Dockerfiles, and Bash scripts.

Instead of writing .yml DSLs interpreted by some CI system, you write real code. Instead of writing ad-hoc Dockerfiles and pushing/pulling images, you chain thunks and share them as code. Instead of writing Bash scripts, you write Bass scripts.

Bass scripts have limited access to the host machine, making them portable between dev and CI environments. They can be used to codify your entire toolchain into platform-agnostic scripts.

In the end, the purpose of Bass is to run thunks. Thunks are serializable command recipes that produce files or streams of values. Files created by thunks can be easily passed to other thunks, forming one big super-thunk that recursively embeds all of its dependencies.

Bass is designed for hermetic builds but it stops short of enforcing them. Bass trades purism for pragmatism, sticking to familiar albeit fallible CLIs rather than abstract declarative configuration. For your artifacts to be reproducible your thunks must be hermetic, but if you simply don't care yet, YOLO apt-get all day and fix it up later.

For a quick run-through of these ideas, check out the Bass homepage.

how does it work?

To run a thunk, Bass's Buildkit runtime translates it to one big LLB definition and solves it through the client API. The runtime handles setting up mounts and converting thunk paths to string values passed to the underlying command.

The runtime architecture is modular, but Buildkit is the only implementation at the moment.

start playing

• prerequisites: git, go, upx

$ git clone https://github.com/vito/bass
$ cd bass
$ make -j install

Bass runs thunks with Buildkit, so you'll need buildkitd running somewhere, somehow.

If docker is installed and running Bass will use it to start Buildkit automatically and you can skip the rest of this section.

Linux

The included ./hack/buildkit/ scripts can be used if you don't have buildkitd running already.

$ ./hack/buildkit/start # if needed
$ bass ./demos/go-build-git.bass

macOS

macOS support works by just running Buildkit in a Linux VM.

Use the included lima/bass.yaml template to manage the VM with limactl.

$ brew install lima
$ limactl start ./lima/bass.yaml
$ bass ./demos/go-build-git.bass

Windows

Same as Linux, using WSL2. Windows containers should work once Buildkit supports it.

editor setup

• vim config: bass.vim

Plug 'vito/bass.vim'

lua <<EOF
require'bass_lsp'.setup()
EOF

cleaning up

The Buildkit runtime leaves snapshots around for caching thunks, so if you start to run low on disk space you can run the following to clear them:

$ bass --prune

the name

Bass is named after the 🔊, not the 🐟. Please do not think of the 🐟 every time. It will eventually destroy me.

rationale

motivation

After 6 years working on Concourse I felt pretty unsatisfied and burnt out. I wanted to solve CI/CD "once and for all" but ended up being overwhelmed with complicated problems that distracted from the core goal: database migrations, NP hard visualizations, scalability, resiliency, etc. etc. etc.

When it came to adding core features, it felt like building a language confined to a declarative YAML schema and driven by a complex state machine. So I wanted to try just building a damn language instead, since that's what I had fun with back in the day (Atomy, Atomo, Hummus, Cletus, Pumice).

why a new Lisp?

I think the pattern of YAML DSLs interpreted by DevOps services may be evidence of a gap in our toolchain that could be filled by something more expressive. I'm trying to discover a language that fills that gap while being small enough to coexist with all the other crap a DevOps engineer has to keep in their head.

After writing enterprise cloud software for so long, it feels good to return to the loving embrace of (((one thousand parentheses))). For me, Lisp is the most fun you can have with programming. Lisps are also known for doing a lot with a little - which is exactly what I need for this project.

Kernel's influence

Bass is a descendant of the Kernel programming language. Kernel is the tiniest Lisp dialect I know of - it has a primitive form beneath lambda called $vau (op in Bass) which it leverages to replace the macro system found in most other Lisp dialects.

Unfortunately this same mechanism makes Kernel difficult to optimize for production applications, but Bass targets a domain where its own performance won't be the bottleneck, so it seems like a good opportunity to share Kernel's ideas with the world.

Clojure's influence

Bass marries Kernel's semantics with Clojure's vocabulary and ergonomics, because you should never have to tell a coworker that the function to get the first element of a list is called 🚗. A practical Lisp should be accessible to engineers from any background.

is it any good?

It's pretty close.

I'm using it for my projects and enjoying it so far, but there are still some limitations and rough edges.

project expectations

This project is built for fun and is developed in my free time. I'm just trying to build something that I would want to use for my own projects. I don't plan to bear the burden of large enterprises using it, but I'm interested in collaborating with and supporting hobbyists.

how can I help?

Try it out! I'd love to hear experience reports especially if things don't go well. This project is still young, and it only gets better the more it gets used.

Pull requests are very welcome, but this is still a personal hobby so I will probably push back on contributions that substantially increase the maintenance burden or technical debt (...unless they're wicked cool).

For more guidance, see the contributing docs.

thanks

• John Shutt, creator of the Kernel programming language.
• Rich Hickey, creator of the Clojure programming language.
• The Buildkit project, which powers the default runtime.
• MacStadium, who have graciously donated hardware for testing macOS support.

bass's Issues

document memo/unmemo

figure out long-term design for cache mounts

• should naked paths really be treated as caches? maybe a 'cache root' would be more appropriate now that path roots are a thing?
• figure out how to express locking semantics (buildkit supports 'locked' for synchronized access or 'shared' for concurrent - what is Bass setting?)

lsp: opening a file that runs an infinite loop will hang lsp

d'oh

should probably not do that anymore

idea: lexically analyze and recurse through any (use) or (import) calls

rename name

path-name?

error go installing with command from website

$ go install github.com/vito/bass/cmd/...@latest

go install github.com/vito/bass/cmd/...@latest: github.com/vito/[email protected]
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

I'm on go version go1.16 linux/amd64 on ubuntu 20.04.

Bass looks awesome, thank you for making it!

rename arg

too useful of a binding for regular code

figure out what should be in stdlib vs a separate module

should stdin also support protocols?

It's kinda weird that it's just straight up JSON.

Related question: should Bass scripts support reading/writing protocols other than JSON?

macOS support (via Linux VM)

Bass uses Buildkit as its default runtime, but it's Linux only. Bass itself is easy to compile and run on macOS (the nightly release already includes binaries for it), but it will need to talk to a Buildkit server over TCP.

I think this could be done using Lima but I don't currently have access to a Mac to dev/test this on.

Also need to make the Buildkit address configurable.

• write Lima config, have bass autodetect buildkitd
• detect appropriate platform via remote buildkitd
• update README
• add bass --prune

rename (in-image) to (with-image)

Fancy console UI should expand all output at the end

Looks like it's keeping the output limit til the very end.

support cache mounts

Cache mounts would help alleviate pain from fetching dependencies, hitting rate limits and just plain ol' taking time.

Buildkit supports this; you just have to provide a cache ID. What should this value be? What is its scope? We obviously want it to span multiple runs, but how do we do that while preventing collisions?

A modest proposal: given that these are effectively global IDs, we could treat it like a global filesystem. When a local path is given to with-mount, it could be conceptualized as a path in a logical global cache filesystem. Under the hood it would just be a string passed to buildkitd that happens to look like a filesystem path. This approach keeps the interface just (with-mount thunk path fspath) instead of calling it a 'cache'.

support reading a file into memory

Use case: storing planned GitHub release title as a file in the repo and passing it to gh release create --title.

There's precedent for this in the (yaml-decode) builtin.

capturing the entire stdout stream as one response value

I would like to be able to run a command and collect its entire stdout as one big string so that I can print it even if the thunk is cached.

Use case: generating coverage reports using CLIs which print human-friendly colorful output. If I run it, accidentally clear/truncate the output, and run it again, I'd like to see the coverage report printed. I could do this by setting the response protocol to :raw or something, which would return the output as the first response value, which could then be (print)ed by the script.

Side task: does it make sense for this to also change stdin to raw? i.e. thunk stdin should contain strings to write verbatim to stdin?

allow local paths to be passed to thunks

Right now Bass is strongly opinionated in preventing any access to the host filesystem (see #6).

With the recent pivot to an image building centric workflow in 9b78421, there is no longer a dependency on image registry push/pull. I think it'd be nice to extend that to local code so that I don't have to repeatedly git push as part of the feedback loop.

The obvious limitation will be that thunks built from local data will not be able to be published in JSON form, but that's not really a goal for local iteration anyway. As long as the failure mode is obvious I don't see why we couldn't just support both.

Caching should be aggressive and accurate so that only changes to files I care about will bust the cache. (.bassignore).

Access to local paths should be read-only (or copy-on-write), and the patterns for accessing it should encourage safety, enforcing it to whatever extent is possible. Untrusted Bass scripts should not be able to easily read arbitrary files.

There should be a clean practice for re-using Bass scripts for building from local code + reproducible code in CI.

docs dont cover *dir*, *env*, (main)

make sure documentation clearly covers thunk output semantics

only the output directory is captured; modifications to inputs do not propagate (mainly affects chaining (from))

see comments in #32

add system for tracking unfriendly errors

• add an interface for friendly errors to implement
• when a non-friendly error bubbles up to bass.WriteError, suggest reporting an issue, or help automate it

publish a vim plugin

• implement a language server, bass-lsp
• vim syntax
• treesitter parser?
• wires up bass-lsp

Until then, here's the config I'm using - it basically hijacks the Clojure config, so you may want to disable that:

lua <<EOF
local configs = require('lspconfig/configs')
local util = require('lspconfig/util')

configs.bass = {
  default_config = {
    cmd = { 'bass-lsp' },
    root_dir = util.root_pattern('.git'),
    filetypes = { 'clojure' }
  },

  docs = {
    description = [[https://github.com/vito/bass]],
  },
};
EOF

autocmd! BufEnter *.bass setlocal ft=clojure lispwords+=job,def,op,defop,defn,defcmd,provide

fix up sneaky dependency on golang image for building shim

I whipped up the shim builder really quickly and never closed this loop - it probably shouldn't just go and fetch that image, it should be more controlled. Maybe it should even be pre-built and mounted in.

more intuitive progress output

Right now the Buildkit runtime just forwards status updates directly through to progrock (the types are the same), which is easy but some things will probably seem confusing to the end-user.

• shim building is noisy
• shim executable wrapping the command will look confusing
• exporting to client isn't super clear
• probably some other stuff

Looks like I can use use llb.WithCustomName to fix some of this.

figure out how to share code

Now that I'm using Bass in two projects (Bass + Booklit) I'm finding myself wanting to reuse a lot of code.

How should this be designed?

It's already possible to fetch libraries and load them using a thunk, but I'd like to not have to write that code multiple times. So it should probably be built in to Bass, maybe as part of stdlib.

safe secrets

Need a way to pass secrets into bass scripts in a way that keeps them safe.

• Decide the default approach for obtaining secret values
• Artifact thunks should not contain secrets in plaintext when encoded to JSON
• Consider censoring known secret values from the script's logs
• Try to prevent footguns with secret usage

Risks to mitigate:

• Committing plaintext secrets into source control
• Printing plaintext secrets to the console UI
• Leaving plaintext secrets in shell history

Goals to consider:

• Easy to scope secrets to workspaces/projects/something
• Fetching from a secret manager (e.g. Vault)
• Supporting interval-based credential rotation without it resulting in mass cache invalidation
• Fetching secret values as close to usage time as possible to support short expirations

Non-goals:

• Don't get too opinionated about best practices - if the barrier to entry is too high, folks may take shortcuts are less safe than a safe-but-not-100%-perfect workflow.

rename (values)

publish artifact .json files in bass releases

*stdin* and *stdout* look weird in docs

scopes should print with keyword keys

Question: is there a method to mount some files/directories from current working directory into the container?

I'm testing the ability of bass to do aggressive caching on a large project that need to be compiled multiple times, and within one step of the compilation, minor predictable changes need to be applied (some constants in a certain file needs to be changed every time). I'm thinking of generating the required files locally, then mounting the files into the container with with-mount <trunk> ./local-dir ./src) but it seems to result in a empty directory being mounted. Is there a method to achieve this, or by design bass require a fixed source from a fixed input?

BTW, this is my first time writing that much Lisp, feel free to point out if I've done something wrong.

add thunk? predicate

godocs, once things stabilize

I'd like everything to have solid godocs, but I've been skipping this until ideas start to settle so I'm not wasting too much time.

nicer Thunk.String

drakeno: <thunk: 946a7e4c59c7ed13cb1e33e78eb201f7cbf715da>
drakeyes: <thunk: (.load "alpine/git")>

Unable to install with go get

Excited to kick the tires on bass!

As per the instructions on the home page, I attempted to install bass with the go get command and was met with an error. I'm not really familiar with Go these days so perhaps I did something wrong.

$ go install github.com/vito/bass/cmd/...@latest
go: downloading github.com/vito/bass v0.0.0-20220103020353-a43629f9cd39
go install: github.com/vito/bass/cmd/...@latest (in github.com/vito/[email protected]):
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

I'm on macOS Big Sur 11.6.2 and am happy to provide other info or try out something else to lend a hand with troubleshooting.

Thanks,
+Jonathan

rename `stream` to `list->source`

primarily to free up 'stream' for local bindings

put extra thought into stdlib

Right now it's entirely ad hoc. Obviously there are no guarantees pre-1.0, but it's worth taking a step back before shipping anything.

swap make-secret arg order

consider just disabling the console UI

I find myself disabling it all the time. It's pretty obnoxious and doesn't show me as much output in a small terminal.

does running a thunk in an absolute path work?

runtimes/command.go L211 loops to make paths relative to the thunk's working directory, but it has a stop condition on dir != "." which seems to assume dir is always relative, and might go into an infinite loop otherwise

deterministic env var order

Right now if you run a thunk with multiple env vars multiple times it will not be cached properly as the order of the env vars may change each time.

ability to bump/un-pin something in bass.lock

lsp should work with cached image resolution and thunk path loading

related: #46

in-image should support ref syntax

Right now it's just turned into {:repository str} so you have to set the full image info:

(from {:repository "golang" :tag "1.17"})

It would be nice if you could do this instead:

(from  "golang:1.17")

consider renaming applicative to function everywhere

for symmetry

op -> operative
fn -> function
defop -> define operative
defn -> define function

set up a proper domain

• wait for DNS changes to propagate
• finish verifying domain
• wait for a certificate to be issued
• enable HTTPS
• update references to vito.github.io/bass

remove (path) in favor of (subpath)

too commonly used as a local var name

also slightly weird that it exists at all

lsp: errors with unbound symbol `*args*`

error! call trace (oldest first):

  5. /home/vito/src/booklit/ci/build.bass:1     (def [ref version] (case *args* () ["...
  4. /home/vito/src/booklit/ci/build.bass:4     (case *args* () ["HEAD" "0.0.0-dev"] ...
  2. (2 internal calls elided)
  1. /home/vito/src/booklit/ci/build.bass:4     *args*

unbound symbol: *args*

Should just bind it to [].

remove the 'ground' binding

document secrets

thunk images should be pinned to digests in a safer manner

Loose image references would be a common way to introduce instability into your build graph.

Currently image references are resolved at the runtime layer, and the digests are cached in ~/.cache/bass/refs.db to prevent hitting Docker Hub's aggressive rate limits.

This makes thunks not as reproducible as they could be, because their image may resolve to a different digest on a different machine or at a different time (if refs.db is cleared).

Some early ideas:

• instead of storing ref digests in refs.db, store them in some sort of bass.lock file local to the repo
• instead of resolving digests at the runtime layer, resolve them ahead of time to ensure thunks are always pinned
• have the runtime return the set of digests it resolved, somehow

Ideally thunks would have digests embedded as soon as possible, because some scripts emit them directly to *stdout* without even running them.

Trade-offs to consider:

• Bass doesn't technically know anything about image resolution, so doing it ahead-of-time might mean changing that.
• Relying on the runtime layer to update .lock feels like it pokes holes through several abstraction layers. But I guess that's just a code problem; it could be abstracted away behind a callback or something.

implement .lock files for image digests and .git ls-remote

Right now digests save in a refs.db file somewhere in the user's home directory (I don't remember which XDG dir it is).

I think it'd be nicer to turn that into a .lock file local to the repository that can be committed and pushed to track exact versions.

• how do we figure out where the file is/should be created?
• what goes into the .lock file? (image digests? libraries?)
• how are dependencies removed?
• how are dependencies bumped?

update help text

that was like one of the first pieces of code ever written and it is not up to par lol