otolonetworks/rebar3_osv

Turn an Erlang/OTP application into a Unikernel image

rebar3_osv

Turn an Erlang/OTP application release into a OSv-based Unikernel image (with a few notable exceptions).

Use

Add the plugin to your rebar config:

{plugins, [
    { rebar3_osv, ".*", {git, "https://github.com/OtoloNetworks/rebar3_osv", {tag, "OTP-21.0"}}}
]}.

You require an Erlang/OTP installation compatible with OSv. The simplest way to get that is to check out the pre-compiled version from Github:

# An example, use whatever location suites you - reference
# it in the rebar.config!
cd /usr/local/packages
git clone --branch OTP-21.0 https://github.com/OtoloNetworks/otp-osv.git

You need to specifiy a couple of things for relx, such as where the OSv specific ERTS is located. rebar3_osv includes a base OSv image, but you can specify your own if required, as well as tweaking the command line used to start applications, turn on 'verbose' output etc. See below for how

{profiles,
 [
  {osv_build,
   [
    {relx, [
            {dev_mode, false}
           , {include_src, false}
           , {include_erts, "/usr/local/packages/otp-osv"}
           , {system_libs, "/usr/local/packages/otp-osv/lib"}
           ]
    {osv, [
           %% {osv_image, "overidden_base.img"}
           %% , {verbose, true}
           ]
    }]
  }
 ]}.

Then just call your plugin directly in an existing application:

$ rebar3 as osv_build osv generate
===> Fetching rebar3_osv
===> Compiling rebar3_osv
===> Verifying dependencies...
===> Compiling cowboy_example
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /home/rickp/src/demo/cowboy_example/_build/osv_build/lib
          /usr/local/packages/osp-osv/usr/lib64/erlang
          /home/rickp/src/demo/cowboy_example/_build/osv_build/rel
===> Resolved cowboy_example-0.1.0
===> rendering builtin_hook_status hook to "/home/rickp/src/demo/cowboy_example/_build/osv_build/rel/cowboy_example/bin/hooks/builtin/status"
===> Including Erts from /usr/local/packages/otp-osv/usr/lib64/erlang
===> release successfully created!
===> OSv image built: /home/rickp/src/demo/cowboy_example/_build/osv_build/cowboy_example.img
===> Command Line: /start-otp.so /otp/releases/0.1.0/cowboy_example /otp/releases/0.1.0/vm.args /otp/releases/0.1.0/sys.config

To run the resulting image, I use a script like this:

#!/bin/sh
IMAGE=_build/osv_build/cowboy_example.img
qemu-system-x86_64 -m 6G -smp 2 --nographic -gdb tcp::1234,server,nowait \
    -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0,scsi=off \
    -drive file=$IMAGE,if=none,id=hd0,cache=none,aio=native \
    -netdev bridge,id=hn0,br=br0,helper=/usr/lib/qemu/qemu-bridge-helper \
    -device virtio-net-pci,mac=00:11:11:11:11:01,netdev=hn0,id=nic0 \
    -device virtio-rng-pci -enable-kvm -cpu host,+x2apic \
    -chardev stdio,mux=on,id=stdio,signal=off -mon chardev=stdio,mode=readline,default \
    -device isa-serial,chardev=stdio

The base OSv image included with rebar_osv has the monitoring webservice built in on port 8000 of whatever IP address the VM gets by DHCP.

Additional Info

OSv is an open-source Unikernel for various virtual machines, designed to exectute a single application on top of a hypervisor. Designed to run Linux software unmodified, it seems an ideal choice to allow an Erlang/OTP application to be packaged with the bare minimum needed to run on a hypervisor.

There was some work done to allow Erlang to run, which involves some patches to the Erlang ERTS - mainly because OSv does not support the notion of fork(), plus a few of the os_mon items are not supported. Some alterations have been made to run epmd as a thread rather than a separate process. (We want to move to use the erlang native epmd at somepoint). Along with this is the fact that ports will not work, which means some of the os_mon applications do not work properly.

I've taken the patches generated by the OSv contributors (myself included) and applied them to an OTP tree which we maintain on GitHub. We also maintain a compiled version so you do not need to download and compile Erlang/OTP to get going with this.

Limitations for Erlang under OSv

1. rebar3_osv must be run on Linux x86_64, as the release is uploaded by running the image under kvm/qemu and transfering in the release files.
2. Needs to use the modified OTP (currently based on 21.0) and this needs to match that installed on the system
3. Ports are not supported
4. NIFs are supported but may need compiling -fpie (the default on many distributions)

Building your own OSv Image

Building your own OSv image is relatively straightforward. Checkout the OSv code from https://github.com/cloudius-systems/osv and update the submodules.

Then its just a case of compiling:

$ scripts/build -j4 fs_size_mb=200 image=cloud-init,httpserver,openssl,libz,ncurses

If you want a debug image, use:

$ scripts/build -j4 mode=debug fs_size_mb=200 image=cloud-init,httpserver,openssl,libz,ncurses

The fs_size_mb sets the size of the image FS, so if you need more storage, increase the size. The file you need is in the build/*/usr.img - you can use either the debug or the release image, depending on what you need.

Once built, you can reference this image from the rebar3 configuration using the osv_image variable:

           {osv_image, "/home/rickp/osv/build/debug.x64/usr.img"}
           %% , {verbose, true}

How is the final image built

The way the image is built is as follows:

1. A release is generated using the OSv compatible ERTS. As this is generated using the host Erlang install, this probably needs to match the OSv specific ERTS (otherwise the boot scripts may be worng).

2. The osv_image is copied to a new file (called after the application name) and started with the command that starts the cpio application.

3. The release built in stepa 1 is CPIO-ed into the running image, along with a small starter program which takes the place of 'erl' - sets the appropriate environment variables, then loads erlexec and invokes main() to start erlang.

rebar3_osv's Issues

Hot code reloading?

This is a really cool project! I'm using it as part of learning about deploying Erlang.

Could the OSv unikernel concept be compatible with hot code reloading? (I understand it may not be implemented now, just wondering if the concepts could be used together and if there might be a benefit to doing so.)