firecracker-microvm/firecracker-go-sdk

An SDK in Go for the Firecracker microVM API

A basic Go interface to the Firecracker API

This package is a Go library to interact with the Firecracker API. It is designed as an abstraction of the OpenAPI-generated client that allows for convenient manipulation of Firecracker VM from Go programs.

There are some Firecracker features that are not yet supported by the SDK. These are tracked as GitHub issues with the firecracker-feature label . Contributions to address missing features are welcomed.

Developing

Please see HACKING

Building

This library requires Go 1.18 or later and Go modules to build. A Makefile is provided for convenience, but is not required. When using the Makefile, you can pass additional flags to the Go compiler via the EXTRAGOARGS make variable.

Tools

There's a firectl tool that provides a simple command-line interface to launching a firecracker VM. It also serves as an example client of this SDK.

Network configuration

Firecracker, by design, only supports Linux tap devices. The SDK provides facilities to:

• Attach a pre-created tap device, optionally with static IP configuration, to the VM. This is referred to as a "static network interface".
• Create a tap device via CNI plugins, which will then be attached to the VM automatically by the SDK. This is referred to as a "CNI-configured network interface"

CNI

If a VM is configured with a CNI-configured network interface, by default CNI configuration files will be sought from /etc/cni/conf.d and CNI plugins will be sought under /opt/cni/bin (both of these values can be overridden via API fields). CNI network lists must be specified in a configuration file at this time.

It's currently highly recommended to use CNI configuration that includes tc-redirect-tap as a chained plugin. This will allow you to adapt pre-existing CNI plugins/configuration to a tap device usable by a Firecracker VM.

Example

With the following file at /etc/cni/conf.d/fcnet.conflist:

{
  "name": "fcnet",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "ptp",
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.127.0/24",
        "resolvConf": "/etc/resolv.conf"
      }
    },
    {
      "type": "firewall"
    },
    {
      "type": "tc-redirect-tap"
    }
  ]
}

and the ptp, host-local, firewall, and tc-redirect-tap CNI plugin binaries installed under /opt/cni/bin, you can specify, in the Go SDK API, a Machine with the following NetworkInterface:

{
  NetworkInterfaces: []firecracker.NetworkInterface{{
    CNIConfiguration: &firecracker.CNIConfiguration{
      NetworkName: "fcnet",
      IfName: "veth0",
    },
  }}
}

Note that NetworkName in the CNIConfiguration of the API matches the name field specified inside the /etc/cni/conf.d/fcnet.conflist file.

With the above configuration, when the Firecracker VM is started the SDK will invoke CNI and place the final VM inside the resultant network namespace. The end result being:

• Outside the network namespace, a single veth endpoint created by the ptp plugin will exist with a static IP from the host-local plugin (i.e. 192.168.127.1)
  • Users can obtain the IP address and other static network configuration generated for their machine via CNI by inspecting the network interface's StaticConfiguration field, which will be automatically filled out after the machine has been started.
  • The IP address, for example, can be obtained at NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IPAddr after a call to the machine object's Start method succeeds.
• Inside the VM's network namespace:
  • The other side of the veth device will exist with name veth0, as specified by the IfName parameter above, and a different IP (i.e. 192.168.127.2)
  • The tap device created by tc-redirect-tap, which will not have an IP but will have all of its traffic mirrored with the veth0 device
• Inside the actual Firecracker VM guest:
  • A network interface with the same IP as that of veth0 (i.e. 192.168.127.2)
  • Traffic sent on this device will be mirrored with the external veth0 device, so from a practical perspective the VM's internal network interface will externally appear the same as veth0
  • The internal name of the interface is determined by the Guest OS, not the Firecracker Go SDK.

Note that the ptp and host-local plugins are not required, they are just used in this example. The tc-redirect-tap plugin can be chained after any CNI plugin that creates a network interface. It will setup the tap device to be mirrored with the IfName device created by any previous plugin. Any IP configuration on that IfName device will be applied statically to the VM's internal network interface on boot.

Also note that use of CNI-configured network interfaces will require the SDK to be running with at least CAP_SYS_ADMIN and CAP_NET_ADMIN Linux capabilities (in order to have the ability to create and configure network namespaces).

Network Setup Limitations

These limitations are a result of the current implementation and may be lifted in the future:

• For a given VM, if a CNI-configured network interface is specified or a static interface that includes IP configuration is specified, the VM can only have a single network interface, not multiple.
  • Users can specify multiple static interfaces as long as none of them include IP configuration.
• DNS nameserver settings will only be effective if the VM's rootfs makes /etc/resolv.conf be a symlink to /proc/net/pnp.
• Only up to 2 DNS nameservers can be configured within the VM internally.
  • If a static network interface specifies more than 2, an error will be returned.
  • If a CNI-configured network interface receives more than 2 nameservers from the CNI invocation result, the nameservers after the second will be ignored without error (in order to be compatible with pre-existing CNI plugins/configuration).

Questions?

Please use GitHub issues to report problems, discuss roadmap items, or make feature requests.

If you've discovered an issue that may have security implications to users or developers of this software, please do not report it using GitHub issues, but instead follow Firecracker's security reporting guidelines.

Other discussion: For general discussion, please join us in the #general channel on the Firecracker Slack.

License

This library is licensed under the Apache 2.0 License.

firecracker-go-sdk's Issues

Support configuration of the Firecracker MMDS

Firecracker has an internal metadata service that can provide user-supplied data to the VM via an http endpoint. We should support configuration of this service.

Should not require socket path to be set in both firecracker.Config and firecracker.VMCommandBuilder

The SDK currently requires the socket path to be set in two places, and for it to be set to the same thing:

• firecracker.Config
• firecracker.VMCommandBuilder

firecracker.VMCommandBuilder is used for building the command to start the firecracker binary. Specifying the socket path here adds the --api-sock command-line argument that tells Firecracker to open the socket in a particular location.

firecracker.Config is used in the API client so that the client knows how to talk to the Firecracker API.

In order to both start a VM and communicate with it, a user of this SDK is required to specify the socket path in both places. If the user is intending to start a VM and immediately control it (which I think is the most common case), we should make it simpler to do so and allow the field to be specified only once.

Issue new release for new Firecracker version and CNI changes

The SDK could use a new official release that includes the recent upgrade to Firecracker v0.18.0 and the support for CNI-configured network interfaces.

Both those changes are backwards incompatible, which needs to be included in the release notes.

Test each individual handler

In the handlers.go file there are predefined handlers that need to be tested, and since some handlers are optionally ran they may not be tested in our current integration tests. By writing tests for each defined handler allows certainty that the handler behave correctly.

Support forcing CNI network creation even if delete of pre-existing network fails

In reference to the current code's attempt to delete a pre-existing network for a given VM, I've now encountered in the wild a plugin (the firewall CNI plugin) that consistently fails here due to a previous result not being available (which goes against what the CNI spec says).

While we can fix this problem upstream, we should probably also include the ability for users to optionally override failing at that line (perhaps via something like a Force parameter in the CNI configuration) in order to allow use of CNI plugins that aren't perfectly compliant. The default behavior should stay the same though and if the error is overridden with the Force parameter then the error needs to be logged still.

[CI] Failed CI test runs leave dangling tap devices

Failed CI runs leave dangling tap devices, which then prevent retries of the run from succeeding:

ioctl(TUNSETIFF): Device or resource busy

This is because the final command to delete the tap gets skipped if the test fails.

I've been manually fixing this with a pipeline like this:

$ ip tuntap ls | sed -e 's/:.*//' | xargs -n 1 -I{} sudo ip tuntap del {} mode tap

The CI needs to be fixed so that the taps don't dangle after test failures.

API calls take exactly the timeout of time

API calls made through the Unix socket appear to take exactly 500ms, which is the value of firecrackerRequestTimeout. I can observe this by manipulating the timeout in the runtime.ClientOperation.

Update to Firecracker v0.19

There's a new release of Firecracker that has some tweaks to the new vsock API which require updates in the SDK: https://github.com/firecracker-microvm/firecracker/releases/tag/v0.19.0

Buildkite steps occuring on different hosts / difference in execution environments

We're in the process of adding Amazon Linux 2 testing to our CI along with Debian. However, the CI system is misconfigured and running steps randomly across different hosts right now, meaning that dependencies between steps are not captured properly and some environmental differences are showing up and causing failures.

This is impacting #166.

• Constrain builds to only run in our existing Debian environment
• Constrain builds to always run on the same host
• Set up new pipeline for Amazon Linux 2 duplicating the Debian pipeline
• Fix issues caused by environmental differences (install(1) has a different version with different behavior in the -D option)

`make test` fails with "Cannot create vsock device. VhostOpen(VhostOpen(Os { code: 13, kind: PermissionDenied, message: Permission denied }))"

$ make test
⋮
DEBU[0002] PUT /actions HTTP/1.1
Host: localhost
User-Agent: Go-http-client/1.1
Content-Length: 32
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{"action_type":"InstanceStart"}
 
DEBU[0002] HTTP/1.1 500 Internal Server Error
Transfer-Encoding: chunked
Content-Type: application/json
Date: Wed, 05 Dec 2018 21:20:51 GMT

8e
{
  "fault_message": "Cannot create vsock device. VhostOpen(VhostOpen(Os { code: 13, kind: PermissionDenied, message: Permission denied }))"
}
0

 
ERRO[0002] Starting instance: [PUT /actions][500] createSyncAction default  &{FaultMessage:Cannot create vsock device. VhostOpen(VhostOpen(Os { code: 13, kind: PermissionDenied, message: Permission denied }))} 
WARN[0007] firecracker exited: signal: terminated       
⋮

This needs a better error message showing how to fix it.

jailer: flag seccomp-level is not supported anymore

The jailer on the master branch has dropped the --seccomp-level flag here.

The sdk still append this flag to the jailer binary here

Which causes the jailer to fail with error:

Arguments parsing error: Found argument 'seccomp-level' which wasn't expected, or isn't valid in this context.

Support Firecracker rate limiters

Firecracker supports network and disk rate limiters that can help restrict access to physical resources, insulating against noisy-neighbor issues. We don't currently support configuration of these features, but should.

Support retrieval of MMDS from firecracker

There currently is no method on the Machine struct for retrieving metadata from Firecracker.

There's a method to set the metadata and it looks like the client is setup to make the GET call for MMDS, I think things just need to be hooked up between the client and the Machine.

Allow capture of Firecracker logs

Firecracker reports internal logs and metrics using a pair of FIFOs. We support configuration of these FIFOs via the LogFifo and MetricsFifo in firecracker.Config.

We should consider providing an optional mechanism to capture the log stream from the configured FIFO to a file. Persistent logs can be extremely useful when debugging issues with Firecracker.

Add support for PatchGuestDriveByID

the SDK should support PatchGuestDriveByID defined in the swagger.yml file.

Support for tc-redirect-tap in experimental Firecracker frakti runtime

After adding the tc-redirect-tap to the cni configuration, the pod based on docker fails to connect to network. Is it possible to make them usable for both runtime? For the context, I'm using frakti which supports multiple runtime.

Failed setting up FifoLogWriter with jailer when using path other than ./log.fifo

I'm seeing an error when using jailer, if your config has a LogFifo set to something other than ./log.fifo and you use FifoLogWriter you will get a warning message

level=warning msg="setupLogging() returned Failed to open fifo path at \"log.fifo\": open log.fifo: no such file or directory. Continuing anyway."

It looks as though this is due to in jailer.go LogFifo is overwritten with the base of the configured path, which should just be the fifo file. When this happens the captureFifoToFile will try read from the modified LogFifo value which will only look for a file right next to where the executable is running from.

Looking at other PRs looks like we should keep knowledge of jailer out of machine.go so I would think then Jailer should call captureFifoToFile before resetting LogFifo

Support Firecracker 0.18

Fireckracker 0.18 is out! The biggest change is the way Firecracker supports vsock.

https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md

Support CNI-configured network interfaces

(This code resolving this is already out, retroactively creating to have an issue to reference)

As part of the design for adding CNI support to Firecracker-containerd, it was decided that most of the functionality should be added to the Go SDK so it can benefit both Firecracker-containerd and potentially other consumers. The relevant design doc can be found here: https://github.com/firecracker-microvm/firecracker-containerd/blob/master/docs/networking.md

firectl should check PATH for firecracker by default

Currently, if you don't explicitly provide the location of the firecracker binary, firectl fails with an unhelpful error:

$ ./cmd/firectl/firectl --root-drive=testdata/drive-2.img --kernel-opts="ro console=ttyS0 noapic reboot=k panic=0 pci=off nomodules"
INFO[0000] Called startVMM(), setting up a VMM on ./firecracker.sock
ERRO[0000] Failed to start VMM: fork/exec : no such file or directory
FATA[0000] Firecracker Init returned error fork/exec : no such file or directory

We should address this two ways:

1. Consult PATH by default, overriding only if --firecracker-binary= is provided.
2. Report a meaningful error in the event that there is no firecracker binary found or provided.

machine_test.go:250: attaching vsock failed: [PUT /vsocks/{id}][400] putGuestVsockByIdBadRequest &{FaultMessage:Invalid request method and/or path: PUT /vsocks/foo}

$ make test
⋮
DEBU[0002] PUT /vsocks/foo HTTP/1.1
Host: localhost
User-Agent: Go-http-client/1.1
Content-Length: 27
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{"guest_cid":3,"id":"foo"}
 
DEBU[0002] HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Content-Type: application/json
Date: Wed, 05 Dec 2018 22:12:51 GMT

4c
{
  "fault_message": "Invalid request method and/or path: PUT /vsocks/foo"
}
0
⋮
--- FAIL: TestMicroVMExecution (7.05s)
    --- FAIL: TestMicroVMExecution/TestAttachVsock (0.00s)
        machine_test.go:250: attaching vsock failed: [PUT /vsocks/{id}][400] putGuestVsockByIdBadRequest  &{FaultMessage:Invalid request method and/or path: PUT /vsocks/foo}

Needs a better error message.

How to use metrics and logs fifo with the jailer?

Whatever I do, I can't seem to get logs and metrics to work with the SDK. They work fine if I use the API manually.

I get the following:

WARN[0004] setupLogging() returned [PUT /logger][400] putLoggerBadRequest  &{FaultMessage:Could not open logging fifo: Failed to open pipe. Error: entity not found}. Continuing anyway.

The jailer requires putting files in its chrootBasePath + "/root", which makes sense. I don't think this is working with the SDK.

While I'm here, what's the difference between using EnableJailer and using WithProcessRunner(jailer)?

Example program:

	ctx := context.Background()
	vmmCtx, vmmCancel := context.WithCancel(ctx)
	defer vmmCancel()
	devices := []models.Drive{}
	rootDrive := models.Drive{
		DriveID:      firecracker.String("1"),
		PathOnHost:   &"./rootfs",
		IsRootDevice: firecracker.Bool(true),
		IsReadOnly:   firecracker.Bool(false),
	}
	devices = append(devices, rootDrive)
	jailer := firecracker.NewJailerCommandBuilder().
		WithID("my-test-id").
		WithUID(123).
		WithGID(100).
		WithNumaNode(0).
		WithExecFile("./firecracker").
		WithChrootBaseDir("./jails").
		WithStdin(os.Stdin).
		WithStdout(os.Stdout).
		WithStderr(os.Stderr).
		Build(vmmCtx)

	fcCfg := firecracker.Config{
		SocketPath:      "./jails/firecracker/my-test-id/api.socket",
		KernelImagePath: "./hello-vmlinux.bin",
		KernelArgs:      "console=ttyS0 reboot=k panic=1 pci=off",
		Drives:          devices,
		LogFifo:         "./logs.fifo",
		MetricsFifo:     "./metrics.fifo",
		LogLevel:        "Info",
		NetworkInterfaces: []firecracker.NetworkInterface{firecracker.NetworkInterface{
			MacAddress:  "AA:FC:00:00:00:01",
			HostDevName: "tap0",
		}},
		MachineCfg: models.MachineConfiguration{
			VcpuCount:  2,
			HtEnabled:  true,
			MemSizeMib: 256,
		},
	}

	log.Println("Creating VM Machine")
	m, err := firecracker.NewMachine(vmmCtx, fcCfg, firecracker.WithLogger(log.NewEntry(logger)), firecracker.WithProcessRunner(jailer))
	if err != nil {
		return err
	}

	if err := m.Start(vmmCtx); err != nil {
		return err
	}
	defer m.StopVMM()

	// wait for the VMM to exit
	if err := m.Wait(vmmCtx); err != nil {
		return err
	}
	log.Printf("Start machine was happy")
	return nil

(this code might be incomplete)

Yields the following logs:

INFO[0003] Creating VM Machine
DEBU[0003] Called NewMachine()
DEBU[0003] Called Machine.Start()
DEBU[0003] Marking Machine as Started
DEBU[0003] Running handler validate.Cfg
DEBU[0003] Running handler fcinit.StartVMM
INFO[0003] Called startVMM(), setting up a VMM on ./jails/firecracker/my-test-id/api.socket
DEBU[0003] VMM started socket path is ./jails/firecracker/my-test-id/api.socket
DEBU[0003] Setting up signal handler
DEBU[0004] returning from startVMM()
DEBU[0004] Running handler fcinit.BootstrapLogging
DEBU[0004] Created metrics and logging fifos.
2019-04-08T19:04:31.865532096 [my-test-id:ERROR] Received Error on synchronous Put request on "/logger" with body "{\"level\":\"Info\",\"log_fifo\":\"./logs.fifo\",\"metrics_fifo\":\"./metrics.fifo\",\"options\":[],\"show_level\":true,\"show_log_origin\":false}\n". Status code: 400 Bad Request.
WARN[0004] setupLogging() returned [PUT /logger][400] putLoggerBadRequest  &{FaultMessage:Could not open logging fifo: Failed to open pipe. Error: entity not found}. Continuing anyway.
DEBU[0004] Running handler fcinit.CreateMachine

I tried various combinations of manually creating a remove the fifos, but didn't have any luck. I can't control much of the flow with the SDK.

Support customizing tap device uid/gid in tc-redirect-tap plugin

Right now it's not possible for users of tc-redirect-tap to set a specific tap uid/gid to use; it will always be created with the uid/gid of the caller.

We should add support for customizing the uid/gid. We need to decide if it is best added as a CNI_ARG or a parameter in the CNI configuration file.

network configuration is not working

getting error
[PUT /network-interfaces/{iface_id}][400] putGuestNetworkInterfaceByIdBadRequest &{FaultMessage:missing field state at line 1 column 71}
panic: failed to initialize machine: [PUT /network-interfaces/{iface_id}][400] putGuestNetworkInterfaceByIdBadRequest &{FaultMessage:missing field state at line 1 column 71}

api is not getting state

publishing `tc-redirect-tap` in releases

Could you please publish tc-redirect-tap on Releases for the next release, so that users can easily download it when setting up something using Firecracker and CNI.

tests are not killing all firecracker processes

#17 introduced a leaking firecracker process when running tests.

release 0.20.0

Looks like 0.20.0 is mentioned in the changelog but hasn't been tagged yet -- is it ready for a release?

tests: panic when firecraker binary is not found

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x490a0e]

goroutine 13 [running]:
testing.tRunner.func1(0xc00025e300)
	/home/skarp/.gobin/go/src/testing/testing.go:792 +0x387
panic(0x8779e0, 0xc9dbb0)
	/home/skarp/.gobin/go/src/runtime/panic.go:513 +0x1b9
os.(*Process).signal(0x0, 0x9758a0, 0x96df98, 0x0, 0x0)
	/home/skarp/.gobin/go/src/os/exec_unix.go:56 +0x2e
os.(*Process).Signal(0x0, 0x9758a0, 0x96df98, 0xc0004be480, 0x4f542d)
	/home/skarp/.gobin/go/src/os/exec.go:131 +0x3f
github.com/firecracker-microvm/firecracker-go-sdk.(*Machine).stopVMM(0xc000143800, 0x795601, 0xc00013c450)
	/home/skarp/projects/firecracker-go-sdk/machine.go:383 +0xa8
github.com/firecracker-microvm/firecracker-go-sdk.(*Machine).StopVMM(0xc000143800, 0xf, 0xc63015)
	/home/skarp/projects/firecracker-go-sdk/machine.go:377 +0x2b
github.com/firecracker-microvm/firecracker-go-sdk.testStopVMM(0x976a00, 0xc000028118, 0xc00025e300, 0xc000143800)
	/home/skarp/projects/firecracker-go-sdk/machine_test.go:295 +0x2f
github.com/firecracker-microvm/firecracker-go-sdk.TestMicroVMExecution.func12(0xc00025e300)
	/home/skarp/projects/firecracker-go-sdk/machine_test.go:143 +0x46
testing.tRunner(0xc00025e300, 0xc00000c640)
	/home/skarp/.gobin/go/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
	/home/skarp/.gobin/go/src/testing/testing.go:878 +0x353
exit status 2

CNI, ip= kernel argument and multiple interfaces

Linux kernel image we're using has compiled-in support for bonding, so if we use it with Firecracker, it comes up with eth0 and bond0 interfaces by default. When firecracker does CNI, it injects ip= kernel argument (and doesn't allow that argument to be passed in) with final IP assigned to the VM, but it skips specifying actual device name: https://github.com/firecracker-microvm/firecracker-go-sdk/blob/master/cni/vmconf/vmconf.go#L120-L122

In our case this leads to kernel trying to use bond0 which is obviously not configurable.

I have two options here, so asking which one is preferable (I can submit a PR):

1. We use static CNI, so we know IP address in advance, and we can build ip= kernel argument before VM is launched. So first option is to skip setting ip= kernel argument based on config: https://github.com/firecracker-microvm/firecracker-go-sdk/blob/master/machine.go#L392-L396

2. Second option is to add interface name to machine config, and pass it through the layers so that code in vmconf.go can use explicit interface name if configured, or leave it empty as it is today.

Which way looks better for the project?

Make hardcoded timeouts configurable

When attempting to reproduce some rare errors in firecracker-containerd, I've at times increased load quite a bit which has caused me to hit various hardcoded timeouts in the go sdk. When I increase the timeouts in my fork the operations succeed, so the timeouts are just set too low for the tests I'm running. The current values seem fine as defaults, but they should be configurable.

Ones I've encountered:

• firecracker-go-sdk/machine.go

  Line 495 in 9e2ff62

  err = m.waitForSocket(3*time.Second, errCh)

• firecracker-go-sdk/firecracker.go

  Line 29 in 9e2ff62

  const firecrackerRequestTimeout = 500 * time.Millisecond

• firecracker-go-sdk/firecracker.go

  Line 173 in 9e2ff62

  timeout, cancel := context.WithTimeout(ctx, 250*time.Millisecond)

machine_test.go:262: StopVMM failed: os: process already finished

$ make test
⋮
DEBU[0002] PUT /actions HTTP/1.1
Host: localhost
User-Agent: Go-http-client/1.1
Content-Length: 32
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{"action_type":"InstanceStart"}
 
DEBU[0002] HTTP/1.1 204 No Content
Content-Length: 0
Date: Wed, 05 Dec 2018 21:23:39 GMT

 
INFO[0002] startInstance successful: [PUT /actions][204] createSyncActionNoContent  
INFO[0003] firecracker exited: status=0                 
--- FAIL: TestMicroVMExecution (7.02s)
    --- FAIL: TestMicroVMExecution/TestStopVMM (0.00s)
        machine_test.go:262: StopVMM failed: os: process already finished

⋮

MMDS support does not work

#12 introduced support for the Firecracker MMDS service. Although the PR included tests, it did not exercise the functionality end-to-end. It turns out that MMDS support does not actually work. This can be demonstrated using firectl as shown below. The expected behavior, when passing valid json via firectl's --metadata option is that the metadata is available via the 169.254.169.254 HTTP interface, but instead it aborts due to an empty body being passed to firecracker's mmds API.

$ ./firectl --kernel ~/bin/vmlinux --root-drive ~/src/firecracker/images/rootfs.img \
  -d --metadata='{"my_key":"xyz"}' --tap-device=fc-test-tap0/6a:11:48:21:1a:d5
...
DEBU[0000] Running handler fcinit.SetMetadata           
DEBU[0000] PUT /mmds HTTP/1.1
Host: localhost
User-Agent: Go-http-client/1.1
Content-Length: 0
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

 
DEBU[0000] HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Content-Type: application/json
Date: Thu, 28 Feb 2019 19:15:47 GMT

45
{
  "fault_message": "EOF while parsing a value at line 1 column 0"
}
0

 
ERRO[0000] Setting metadata: [PUT /mmds][400] putMmdsBadRequest  &{FaultMessage:EOF while parsing a value at line 1 column 0} 
WARN[0000] Failed handler "fcinit.SetMetadata": [PUT /mmds][400] putMmdsBadRequest  &{FaultMessage:EOF while parsing a value at line 1 column 0} 
FATA[0000] Failed to start machine: [PUT /mmds][400] putMmdsBadRequest  &{FaultMessage:EOF while parsing a value at line 1 column 0} 

Root tests are no longer being run in the CI system

I just noticed that the CI appears to now be skipping root tests even in the "root test" stage. See the most recent build of the master branch (expand "root tests", you'll see several SKIPs that shouldn't be there): https://buildkite.com/firecracker-microvm/firecracker-go-sdk/builds/367#2fcbb91a-b08c-4685-acf0-db72fb4258a5

Two commits prior they were run though: https://buildkite.com/firecracker-microvm/firecracker-go-sdk/builds/316#9285b823-1672-47da-a14a-d1d7eb27ad08

Support firecracker 0.17.0 API

The SDK currently is based on the Firecracker 0.15.x API. The current release is 0.17.0, and there have been some breaking changes to the API specification.

Upgrade to firecracker v0.14.0

Firecracker v0.13.0 has been released: https://github.com/firecracker-microvm/firecracker/releases/tag/v0.13.0

Support the firecracker jailer

Firecracker provides a jailer that enhances the security boundary of the Firecracker process. We don't currently support it at all, but should.

Support customizing tap device name in tc-redirect-tap plugin

Right now it's not possible for users of tc-redirect-tap to set a specific tap device name to use; it will always be created with the name the kernel decides (usually tap0).

We should add support for that. We need to decide if it is best added as a CNI_ARG or a parameter in the CNI configuration file.

Add support for patching rate limiters

The release of v0.15.0 of firecracker introduces a new API which allows the patching of rate limiters. Currently the SDK does not support this, but should.

goroutine within captureFifoToFile may leak

I am under the suspicion that this goroutine may be leaking if the file has been deleted or unlinked. More testing may be needed to verify this.

Race condition when using LogWriter

After #145 was merged there now exists a race condition when setting up a LogWriter

In captureFifoToFile this goroutine

	go func() {
		defer func() {
			if err := fifoPipe.Close(); err != nil {
				logger.Warnf("Failed to close fifo pipe: %v", err)
			}

			if err := syscall.Unlink(fifoPath); err != nil {
				logger.Warnf("Failed to unlink %s: %v", fifoPath, err)
			}
		}()

		if _, err := io.Copy(w, fifoPipe); err != nil {
			logger.Warnf("io.Copy failed to copy contents of fifo pipe: %v", err)
		}
	}()

will not block on io.Copy, I think this is due to opening with syscall.O_NONBLOCK unsure though. This causes the defer to trigger and syscall.Unlink to remove the originally configured log.fifo. When using jailer if this happens before the LinkFilesHandler handler executes you can get a missing file error when trying to hardlink the log.fifo shown here.

If not using jailer I think that io.Copy could just exit and BootstrapLoggingHandler could have an error if it sent it's logging setup request after the syscall.Unlink occurs.

unable to kill go lang process if vm is running in background

i get this when i start vm
INFO[0000] Called startVMM(), setting up a VMM on /tmp/firecracker172.17.0.2.sock
INFO[0000] VMM logging and metrics disabled.
INFO[0000] refreshMachineConfig: [GET /machine-config][200] getMachineConfigOK &{CPUTemplate:Uninitialized HtEnabled:false MemSizeMib:120 VcpuCount:1}
INFO[0000] PutGuestBootSource: [PUT /boot-source][204] putGuestBootSourceNoContent
INFO[0000] Attaching drive /home/[email protected]/Desktop/built.io/deepakWork/firecracker/firecracker-config/hello-rootfs.ext4, slot root-drive, root true.
INFO[0000] Attached drive /home/[email protected]/Desktop/built.io/deepakWork/firecracker/firecracker-config/hello-rootfs.ext4: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent
INFO[0000] startInstance successful: [PUT /actions][204] createSyncActionNoContent
^CINFO[0009] Caught signal interrupt
WARN[0009] firecracker exited: signal: interrupt

and then i am not able to kill ( CTRl + C ) my golang process if vm is running in background.

Support configuration of Jailer binary path

Right now, the default path to the jailer binary is always used when configuring the Jailer through the JailerConfig struct:

firecracker-go-sdk/jailer.go

Line 125 in b0a25b1

return JailerCommandBuilder{}.WithBin(defaultJailerBin)

Instead, the path to the jailer binary should be optionally configurable in JailerConfig and default to the existing path if not specified.

Support additional VM console options

We currently support several console options for the VM, including stdio to use the existing IO streams or xterm to run in a graphical terminal. Additional ones that could be useful are:

1. Arbitrary file -- Log stdout/stdio to an arbitrary file and read stdin from /dev/null
2. Inherited file descriptor. -- Allow the client to provide existing file descriptors for any or all of the standard IO streams.

Maybe others?

Rethink signal handling

Default signal handling in Linux could lead the firecracker process to become orphaned. To prevent this, the SDK sets up some default signal handlers that forward signals along to the firecracker process. However, this is pretty implicit and not super discoverable unless you read the code, and it may interfere with signal handling that the calling application intends to perform. We should think about:

• what we want signal handling to be
• whether the SDK should register signal handlers by default
• how this can be made more explicit and less implicit

Add Firecracker vsock client code

The recent vsock changes in Firecracker v0.18.0 result in some additional requirements on top of the previously-used standard vsock interface (for which there are already 3rd party libraries), see their docs and this PR for example.

Given there is now Firecracker-specific logic to interfacing with vsock, it would make sense for the Go SDK to contain that code rather than have all consumers of the Go SDK be forced to reimplement it themselves.

Missing input validation on firecracker console settings

There are a couple issues with console handling in firectl.

1. There's no sanity checking at all on the value passed to --firecracker-console. E.g.

ulap:firecracker-go-sdk‹master›$ ./cmd/firectl/firectl --firecracker-binary=/usr/local/bin/firecracker --kernel=~/bin/vmlinux --root-drive=~/src/firecracker/images/image-debootstrap.img --kernel-opts="console=ttyS0 noapic reboot=k panic=1 pci=off nomodules rw init=/sbin/init" --vsock-device=root:3 --firecracker-console=xyz
INFO[0000] Called startVMM(), setting up a VMM on ./firecracker.sock 
INFO[0000] VMM logging and metrics disabled.            
INFO[0000] refreshMachineConfig: [GET /machine-config][200] getMachineConfigOK  &{CPUTemplate:Uninitialized HtEnabled:true MemSizeMib:512 VcpuCount:1} 
INFO[0000] PutGuestBootSource: [PUT /boot-source][204] putGuestBootSourceNoContent  
INFO[0000] Attaching drive /home/ANT.AMAZON.COM/nmeyerha/src/firecracker/images/image-debootstrap.img, mode rw, slot 1, root true. 
INFO[0000] Attached drive /home/ANT.AMAZON.COM/nmeyerha/src/firecracker/images/image-debootstrap.img: [PUT /drives/{drive_id}][204] putGuestDriveByIdNoContent  
INFO[0000] startInstance successful: [PUT /actions][204] createSyncActionNoContent  
^CINFO[0001] Caught signal interrupt                      
WARN[0001] firecracker exited: signal: interrupt        
FATA[0001] startVMM returned error signal: interrupt    

I would expect firectl to report an error in that case, and never actually try to spawn a VM. This issue has likely existed forever.

2. If --firecracker-console=file is specified, but no filename is provided, the behavior is confusing:

ulap:firecracker-go-sdk‹master›$ ./cmd/firectl/firectl --firecracker-binary=/usr/local/bin/firecracker --kernel=~/bin/vmlinux --root-drive=~/src/firecracker/images/image-debootstrap.img --kernel-opts="console=ttyS0 noapic reboot=k panic=1 pci=off nomodules rw init=/sbin/init" --vsock-device=root:3 --firecracker-console=file
FATA[0000] Failed to build command: open : no such file or directory 

I would expect firectl to complain clearly about a missing argument. This is new as of #17

Startup Hangs when using LogWriter

This pertains to PR #134 I worked on and it introduced the issue.

When it was suggested to move captureFifoToFile to the handler, I forgot about the fact that if Firecracker is not started before we start listening on the log.fifo then os.Open in captureFifoToFile will block. Since we moved captureFifoToFile to a handler, this blocks the handlers which execute before we start Firecracker so it'll never unblock., shown here.

Mostly creating this issue to spark a discussion on the best place to put captureFifoToFile. The implementation before the above PR did not work with Jailer as the sdk jailer setup would change the path to the LogFifo in the config before we tried to listen on it.

I've outlined workflows from what I understand needs to happen in the case of jailer and plain firecracker.

Firecracker:

• Create log.fifo and metric.fifo
• Start Firecracker
• Configure logging and metrics via API
• Call captureFifoToFile

Jailer:

• Create log.fifo and metric.fifo
• Jailer chroot for vm setup
• Hard link log.fifo and metric.fifo into jailer chroot for VM
• Jailer start Firecracker
• Configure logging and metrics via API
• Call captureFifoToFile on original log.fifo path (not hard linked)

Add support for bind mount and mknod dev/file mapper

The jailer currently supports a naive approach to sharing files in its root drive which is hard linking. We want to move away from this and support bind mounting of files and running mknod on devices

Split out firectl to its own repo

The firectl binary stands on its own, and would help make downstream packaging (namely in Amazon Linux) much less confusing.

Add support for describeInstance

Hi all,

Firecracker-containerd GetVMInfo API doesn't really call the VM API, we want to periodically call some VM API to check if instance is still responsive, we'd like to use describeInstance API for this purpose, but it doesn't seem it's implemented, do you have plan to implement it? Apologize ahead if you already done this.

Thanks,
Haikuo