ottomatica/baker

👨‍🍳🍞 Quick and easy baking of computing environments.

Baker 🍞 |

Meet Baker! -- a simple tool for provisioning virtual machines and containers. With Baker you can quickly create development environments and run your code. With one tool, you have the functionality of vagrant, docker, ansible, and task runners like grunt.

See a running demo below:

For more details, checkout docs.getbaker.io and join our Slack.

Install from source

git clone https://github.com/ottomatica/Baker
cd Baker
npm install
npm link

Also see other binary installation options.

Using Baker

Baker uses a configuration file (baker.yml) in the root directory of you project. This is an example of a baker.yml file. By running baker bake Baker provisions a VM with nodejs installed, and the specified ip address and port forwarding rules configured for you. You can access the VM by running baker ssh or run commands inside the VM with baker run <Command Name>. Your code is accessible in the VM via a shared folder.

---
name: baker-test
vm:
  ip: 192.168.22.22
  ports: 8000
lang:
  - nodejs9
commands:
  serve: cd /baker-test/deployment/express && npm install && node index.js

You can also point to a git repository with a baker.yml file, and and Baker will do the rest:

$ baker bake --repo https://github.com/ottomatica/baker-test.git

Baker also supports creating environments inside containers that do not require a VM.

name: baker-docs
container: 
  ports: 8000
lang:
  - python2
commands:
  build: mkdocs build
  serve: mkdocs serve -a 0.0.0.0:8000
  gh-deploy: mkdocs gh-deploy

Setting up a Java environment with MySQL can be done easily.

name: onboard
vm:
  ip: 192.168.8.8
  ports: 8080
vars:
  - mysql_password:
      prompt: Type your password for mysql server
tools:
  - maven
services:
  - mysql:
      version: 8
      service_conf: env/templates/mysql.cfg
      client_conf: env/templates/my.cnf
lang:
  - java8
config:
  - template: 
      src: env/templates/hibernate-template.cfg.xml 
      dest: /Onboarding/CoffeeMaker/src/main/resources/hibernate.cfg.xml
commands:
  serve: cd CoffeeMaker && mvn spring-boot:run
  debug: cd CoffeeMaker && mvnDebug spring-boot:run
  test: cd CoffeeMaker && mvn test

baker's Issues

baker.yml version 2

• Dockerfile captures the runtime of a process But it does not capture the dev toolchain + other components like CI
• terraform captures cloud resources and environments settings but doesn't deal with configuration.
• vagrant captures small scale local environments, but doesn't translate to remote environments, nor necessarily deal well with clusters + remote components like CI (edited)
• TravisCI uses conventions to minimize the specification needed for a CI env, but you can't use that for local dev or remote.

Want to support a simple specification like the following, and let baker do the rest:

tools:
  - jenkins-job-builder
services:
  - jenkins
clusters:
  - nomad

Add support for requesting another baker vm's ssh_key and storing in baker vm.

This would be useful for doing something like an ansible setup with a client vm, or eventually can be useful for clusters.

Something like:

keys:
  - checkbox
  - checkbox-db

And this would copy the private_key and store in /keys/checkbox, etc.

Implement Vault Feature

Allow checkout of keys from a vault.

User adds this to baker.yml

bake:
  ansible:
    source: src/env/
    playbooks:
      - bootstrap.yml
      - configure.yml
  vault:
    source: src/env/vault.yml
    checkout: 
      key: alt-code-db-key
      dest: ~/.ssh/alt-code_rsa

User is prompted to enter password to checkout key from vault, which is copied to destination on VM.

Now, key is copied to VM:

Run npm install if package.json in repo and node.js bakelet is used.

⚙️ Rename bakerlet=>bakelet

• Rename bakerlets-source repo.
• Rename directory references.
• Update tests.
• Update docs.

Automatically get the correct version when packaging for macOS, Windows, Linux

Update the packaging scripts to get the correct version from package.json. Now updating 3 files manually for each release ☹️.

The packaging scripts might need some refactoring too (specially for Linux).

Baker should bring up control machine if it is down

Baker website

⚙️ Create updated ascii-video, use on site.

• Should finish #1 first, so we can run without ansible playbook output.
• Create video.
• Update docs.

Fix error message when vm name doesn't exist for `baker up`

Expect it to say "cannot find machine 'teststested16'. Please check status by running baker status.

Instead, get:

cjparnin:Dazed-paper gameweld$ baker up teststested16
events.js:163
      throw er; // Unhandled 'error' event
      ^

Error: spawn vagrant ENOENT
    at exports._errnoException (util.js:1034:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
    at onErrorNT (internal/child_process.js:367:16)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
    at Module.runMain (module.js:607:11)
    at run (bootstrap_node.js:423:7)
    at startup (bootstrap_node.js:147:9)
    at bootstrap_node.js:538:3

Baker2 format doesn't handle missing variables section.

local ansible mode

I'm thinking of ways we could eliminate having a big vm for ansible control machine. Since baker VMs are going to be always Linux, they why not install ansible on those VM behind the scene and provision localhost?
I think the ansible vm that we have is working very nicely but this might worktoo. Each approach has its own pros and cons

I think a good way to explore this is to include an option in the baker.yml to use a flag for local ansible mode. if it has that flag, then set up things so that it does it through local ansible

large ram size warning

1. Should not be X
2. Should probably be yellow
3. Think default threshold can be slightly higher.

Investigate package support with baker2 format.

mongodb Bakelet

brew install failure

Expected behavior

according to Mac instructions found here:

If you have homebrew installed, Simply run this in your terminal:

brew install ottomatica/ottomatica/baker --devel

Actual behavior

==> Tapping ottomatica/ottomatica
Cloning into '/usr/local/Homebrew/Library/Taps/ottomatica/homebrew-ottomatica'...
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 0), reused 1 (delta 0), pack-reused 0
Unpacking objects: 100% (5/5), done.
Tapped 1 formula (29 files, 23.7KB).
==> Installing baker from ottomatica/ottomatica
Warning: ottomatica/ottomatica/baker 0.3.0 is available and more recent than version macos.
==> Downloading https://github.com/ottomatica/baker-release/releases/download/latest-dev/baker-macos-latest.tar.gz

curl: (22) The requested URL returned error: 404 Not Found
Error: Failed to download resource "baker"
Download failed: https://github.com/ottomatica/baker-release/releases/download/latest-dev/baker-macos-latest.tar.gz

Operating system

macOS High Sierra v10.13.6

Baker version

v0.3.0

More complex use case -- suggesting potential architectural changes

More complex setups will require that the ansible may need some additional dependencies.
For example:
https://github.com/alt-code/PeerParity/blob/master/src/db/db.yml

1. A python script needs to be run in order to generate some data.
2. A db needs to be setup which requires mysqlclient connector.

Each of these could conflict with other environment settings. This suggest that we may ultimately want to isolate each VM and its corresponding ansible server instead of sharing multiple ones. Alternatively, we may also consider embedding a local ansible instance within the client VM so that they can be the one that runs local steps. This may be simpliest option.

Subject of another issue: Getting data like rsa keys...

"init" command to create a baker.yml file

baker --ssh should use system ssh client if possible

While javascript ssh is okay, it can be a little flaky with long-term use, it is probably better to rely on the system ssh if it is installed and only fallback to the javascript ssh if necessary.

Python bakerlet improvements

• Add python 3
• Should install pip by default.
• If someone says python, instead of python2 should just default to python2.
• If a repo has a requirements.txt inside, should run sudo -H pip install -r requirements.txt

Create R bakerlet, with packages and system_packages support.

system_packages: libxml, ...
lang:
   - R:
       packages: tidyverse, xml, etc

Introduce easier port expose options under vm

vm:
  ports: 8000, 3000:9000

git (SCM) Bakelet

Missing package.json?

Describe the bug
npm link fails because of missing package.json

vagrant reload and ssh-keyscan

1. vagrant reload if baking a vm if already up.
2. Need to run a local ansible command with something like this:

- name: Get host signatures and add to known_hosts.
  command: ssh-keyscan -H {{item}}
  with_items: "{{ droplet_details.results| map(attribute='droplet.ip_address') | list }}"
  register: known_hosts_keys

#- debug: msg="Signatures are {{ known_hosts_keys.results | map(attribute='stdout') | list}}"

- name: Register hosts to known_hosts
  known_hosts:
    name="{{ item.0 }}"
    key="{{ item.1 }}"
    state=present
  with_together: 
    - "{{ droplet_details.results| map(attribute='droplet.ip_address') | list }}"
    - "{{known_hosts_keys.results| map(attribute='stdout') | list }}"

More graceful restart issues

• If killing baker, ansible server may still have scripts running. Leading to issues like dpkg getting locked. One solution is to basically do a killall on any previous ansible task. We might be able to pattern match processes from ps that look like old ansible tasks.

Explainable configuration error messages.

brew install baker

A few use cases for Baker and Baker Cloud

For a couple of projects we have several resources that are more cloud or server based. For example, VersionBot and the alt-code-db.

In a current project, I have a student that will be creating new tables. I'd like to manage that configuration with ansible. But again, don't want student to have to go through trouble of figuring out how to make that work.

Instead, I want a simple baker command that will help run ansible commands for configuring remote resources. Now, this is a perfect use case for Baker Cloud.

Baker Cloud with Vault services

Further, there are some things I wish I could just keep in Baker Cloud. Like inventories, keys, and vaults. That way that stuff doesn't have to be shared with devs, but can just be held in cloud.

Hosted ansible allows us to add values like easier ansible use and simplifying what needs to be stored in repos.

Provision remote servers with Baker

Some upgrades to the default Vagrant template

Suggestions here:
https://datasift.github.io/storyplayer/v2/tips/vagrant/speed-up-virtualbox.html

Exploring Baker for Windows

update grunt task to create a release on GitHub and upload installers

Use int-test to create a new grunt task. if it passed, build and create a release on GitHub

Sensible defaults for VM.

Ideally would like just a simple defaults to work, requiring really only name.

Installers (+ code signing certificates)

Windows:
https://www.npmjs.com/package/msi-packager

neo4j bakelet

Create based off of:
https://github.com/EricHorton/neo4j-tests/blob/master/neo4j.yml

lxd support

• 1. Create lxd bakelet (under services).
• 2. Use and test with autograde setup.
• 3. Create a command to use lxc client (quick-bake)

Renaming options

--install => --setup
--script => --local

Improve error message if `--install` is not run

Right now does this:

eb2-3270-mac01:dev cjparnin$ ~/projects/Baker/src/baker-macos --repo https://github.com/alt-code/PeerParity
Cloning into 'PeerParity'...
remote: Counting objects: 55, done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 55 (delta 11), reused 35 (delta 7), pack-reused 0
Unpacking objects: 100% (55/55), done.

/snapshot/src/baker.js:234
                throw `==> Couldn't get private ssh key of new VM: ${err}`; // err
                ^
==> Couldn't get private ssh key of new VM: A Vagrant environment or target machine is required to run this
command. Run `vagrant init` to create a new Vagrant environment. Or,
get an ID of a target machine from `vagrant global-status` to run
this command on. A final option is to change to a directory with a
Vagrantfile and to try again.

env variable Bakelet

Baker cluster support

It has been a long-time coming for baker to support a cluster (multiple nodes). Here are some use cases and proposed formats:

• Nomad cluster

We have several projects making use of nomad. Nodes need docker, master node with consul.

cluster:
  - nomad: 
      master:
        services:
          - consul
      nodes[3]:
        services:
          - docker

• Spark cluster

Spark is a pretty useful approach for doing analysis that might need very large amount of RAM.

cluster:
  - spark: 
      nodes[10]:
        memory: 4GB
        cpu: 2

• Plain clusters

For our git mining projects, it is useful to have multiple worker nodes to do a task.

cluster:
  - plain: 
      nodes[10]:
        lang:
          - nodejs9
        services:
          - docker

• Other

It could be useful to have the ability to attach an ad-hoc node the cluster, especially, for things like logging, data collection, etc.

     satellite:
        services:
          - log

Wireframes for Baker Cloud / App

Baker should not hard-code git clone error

https://github.com/ottomatica/Baker/blob/master/src/lib/modules/cloneRepo.js#L15

This hard-coded error message is incorrect in other instances, for example, with invalid authentication credentials.

verbose option

Can turn on things like, vagrant up progress, if verbose is on:
https://github.com/ottomatica/Baker/blob/master/src/baker.js#L239

Simplify baker2 private_network format.

We want to go from

  network:
    - private_network:
        ip: "192.168.31.33"

to

ip: 192.168.31.33

Vagrantfile is not updated

If a setting is changed in baker.yml, not set in Vagrantfile after first creation.

Update commands to use VM info from baker.yml in cwd

Update commands to make specifying VM names optional. By default Baker should use VM info from baker.yml in cwd (if available).

Commands:

• bake
• destroy
• up
• ssh
• halt

Thinking about the package lifecycle.

Scenario. A class wants to get a vm, previously packaged by someone. Maybe it is available online via an url.

• How do those people get the vm?
• How does the code/repo associated with it get connected?
• Is this something that only can be specified in baker.yml
• How do we take advantage of the previously baked things, while still being able to support adding new things in baker.yml or updating something again in the future.
• ...

Default bake command

Instead of having to type baker bake --local ., the bake command should only require baker bake if there is a baker.yml in current working directory and no other parameters provided.

Docker image / build server for making windows installer