edera-dev/krata

krata is an isolation engine for securing compute workloads

krata's Issues

feature: OCI progress should give much more feedback and be cleaner

The current OCI progress implementation is a little jank, which is primarily because it heavily relied upon the assumption that we would have exact progress. With the reimplementation of the OCI assembler, we no longer have exact progress. We can fix this by implementing a more detailed OCI progress layer that can send messages and indicate spinners rather than progress bars.

feature: implement guest exec support

Implement support for exec'ing into a guest, with stdin/stdout/stderr proxied over IDM.

feature: more human friendly output for IDM snooping

dbus-monitor for example looks like this:

signal sender=org.freedesktop.DBus -> dest=:1.102 path=/org/freedesktop/DBus;      interface=org.freedesktop.DBus; member=NameAcquired  string ":1.102"
signal sender=:1.63 -> dest=(null destination) path=/org/gnome/Tomboy/RemoteControl; interface=org.gnome.Tomboy.RemoteControl; member=NoteAdded
   string "note://tomboy/0ae6b0b3-9394-44e9-ad4a-46726e68cebc"
signal sender=:1.63 -> dest=(null destination) path=/org/gnome/Tomboy/RemoteControl; interface=org.gnome.Tomboy.RemoteControl; member=NoteSaved
   string "note://tomboy/0ae6b0b3-9394-44e9-ad4a-46726e68cebc"
signal sender=:1.63 -> dest=(null destination) path=/org/gnome/Tomboy/RemoteControl; interface=org.gnome.Tomboy.RemoteControl; member=NoteDeleted
   string "note://tomboy/0ae6b0b3-9394-44e9-ad4a-46726e68cebc"
   string "New Note 1"

We should have a similar formatting option for our IDM messages.

feature: implement VFS for synthetic inode data when packing/unpacking images

When krata performs unpacking operations, it should do so in a deprivileged context. However as a result we hit issues like #69 which can only be solved with VFS in a deprivileged context.

The best way to handle this is to write the files out to a scratch volume as we do today, but keep track of the inode data ourselves as an overlay. There is likely a crate which could do this for us already, but I haven't gone looking yet.

dmesg pre-init items of possible note

Opening an issue just to track the pre-init dmesg items I spot that may be hints to other issues or just worth addressing at some point

• ACPI: MPS support code is not built-in, using acpi=off or acpi=noirq or pci=noacpi may have problem
• DMI not present or invalid
• tsc: Fast TSC calibration failed
• Kernel parameter elevator= does not have effect anymore. Please use sysfs to set IO scheduler for individual devices
• 

feature: enhance fault tolerance using supervision

Most, if not all of the components within krata use a controller / processor strategy, where a background green thread performs processing and another component can interact with that processor using a controller, which is generally implements Clone.

It would be very wise for us to utilize a supervisor model, so that krata becomes more fault tolerant.

feature: security labels for guests

In order to allow for more expressive security policies, Flask-style (as seen in SELinux/TrustedBSD/Solaris) security labels should be applicable to guests, including default-inherited ones set by the hypervisor.

feature: verify sigstore-based image signatures if present

Although we need a larger policy-controller-like hook mechanism, we should include support for sigstore signatures as part of the "included batteries."

There should also be a configuration option to require signature validation to pass before allowing an image to be ingested.

chore: document security policy for IDM messages between host and guest

We especially want to encode that guests are disallowed from making requests from the host for obvious security reasons. This does not mean that guests might not make some kinds of requests from the host, but the list of allowed requests should be narrow and documented since we consider guests to be untrusted workloads.

bug: krataoci-squashify not preserving setuid bit

On an arch based image:

$ podman run -it --entrypoint=/bin/bash registry.gitlab.com/xenia-group/images/templates/arch-image:main
[root@782eb26dccb3 img]# ls -lah /usr/bin/sudo
-rwsr-xr-x 1 root root 264K Dec 30 18:51 /usr/bin/sudo
[root@782eb26dccb3 img]# 

In the squashfs generated by krata:

$ sudo mount 51c3dd858cb9199a6237a0255b52f9d36ee339eac80ada5adb6c7b8158fa3225.squashfs /tmp/kmnt/
$ cd /tmp/kmnt/
$ ls -lah usr/bin/sudo
-rwxr-xr-x 1 root root 264K Dec 30 18:51 usr/bin/sudo

Seems like the setuid bit is being removed?

feature: kratactl top

Add a top like client to kratactl, which allows users to dig into guests and processes. Once #57 is implemented, add support for sending signals to processes within guests.

ARM64 Support

Support for ARM64 guests is already work in progress, but there are a number of things still to tackle.

Automatic startup of guests on daemon launch.

Implement the ability for guests to launch automatically on startup.

Since krata is a mostly stateless hypervisor, there was originally no intent for automatic startup. However, it may make sense for some use cases to write a simple file that will launch guests automatically.

feature: guest init: add IDM messages for sending signals to child processes

It would be nice to be able to kill a child process in a guest now that we have visibility into the guest process table via IDM.

meta: release, staging, and dev

Tracking issue for concerns around branch and release flow:

• How do we ensure we have control over what is moved into releases?
• How do we ensure our release automation is hardened?
• What should our release cycle look like?

feature: guest metrics support

Implement the ability to read metrics from the guest via the control service.

feature: add IDM snooping utility

It would be useful to have the ability to snoop on the IDM bus, in much the same way as dbus-monitor.

feature: move IDM bus brokering into helper domain

Presently the IDM bus is brokered by kratad on dom0. We should move it into a helper domain.

chore: documentation on custom kernels

Hello,

I am interested in testing krata for use as a runtime for running Talos Linux in Kubernetes. Talos has some unique requirements around the kernel it runs and I was hoping krata could eventually support running custom kernels.

chore: transition IDM request/response names to well-known ones

Presently IDM messages lack a qualified namespace for invoked methods/responses. We should move our responses to something that can be qualified, for example krata.dev/metrics instead of metrics.

feature: transition image packing/unpacking to helper domain

Building on #75, we should move the image packing/unpacking to the proposed helper domain infrastructure to remove it from dom0.

feature: investigate unikernels for some krata services

Some services like the IDM bus, and networking, would be ideal targets to unikernelize.

feature: observability using opentelemetry

Add support to krata for exporting metrics via opentelemetry.

feature: interposable policy decisions via helper domains

It should be possible to have certain kinds of policy decision points flow through IDM, to allow for helper domains to provide pluggable policy engines. Areas where we can add hooks include image ingestion, etc.

You could imagine a helper domain being plugged in to check an image for a valid sigstore signature (#92) or checking an image's SBOM to verify that a blocked component is not allowed to be ingested and ran.

feature: support pulling prebuilt kernels from OCI registries

Presently, a krata user must build a kernel themselves. We can prebuild the kernel instead and publish it in an OCI registry. To do so, we must support fetching kernels from OCI registries, however.

Pass through reasonable TERM environment value

Right now we get weird artifacting on the console when running some software like Powershell:

It is probably because TERM is set to something not very reasonable, like linux.

tracking: OCI runtime conformance

The OCI runtime conformance tests are an exhaustive testsuite which tests behavior of an OCI runtime implementation. We should try to pass as many of the tests as possible, and if necessary, document the rationales for any deviations.

Since OCI targets a shared-kernel runtime architecture we do expect some minor deviations from the normal OCI runtime specs.

feature: move networking backend into helper domain

Krata's default networking backend runs as a process in dom0. Lets convert it to its own helper domain to deprivilege it.

bug: xattrs not being tracked

What happened?

In krataoci-squashify, xattrs are not being tracked.

For example, with the image registry.gitlab.com/xenia-group/images/gnome-arch:main, the file /usr/bin/newuidmap will have the permissions -rwxr-xr-x when it should have -rws--x--x.

Version / Commit

v0.0.9

Relevant log output

No response

feature: gpu passthrough

Implement GPU passthrough support in krata, allowing a guest to access a virtualized unit of a GPU.

feature: reflect object type names in IDM messages

Presently IDM types are implied in responses processed by a user making an RPC over the IDM bus, making working with IDM unnecessarily difficult if you don't have access to the raw protobuf messages.

We should reflect object type names directly, for example, a metric should have a type field with Metric.

docs: contributing guide

Create a contributing guide (CONTRIBUTING.md) to describe how developers can get involved with krata.

tracking: privilegeless kratad

This is a tracking issue to track work items needed to fully deprivilege krata. See the tasklist for the actual work items.

Overall architectural work

• #75
• #79

Disk image management

• #76
• #77

IDM

• #78

Networking

• #80

feature: implement stdin/stdout/stderr multiplexing over IDM

For workloads which do not require a TTY, we should have the guestinit set up stdin/stdout/stderr multiplexing over IDM and use that instead of /dev/console. This allows /dev/console to be kept strictly for kernel logging.

feature: generate default names for containers

We should generate a 2-part name using words from a predefined dictionary that are 3-6 characters long, so people using kratactl can type that instead of the UUID, in case of contexts (like ZelliJ over SSH, cough) that make copy/paste less viable.

I'm unsure if this could be a vector for security issues, but if so, we could have this only occur when a flag is passed, or maybe only on manual launch vs other forms of deployment.

chore: document security policy for intraguest IDM messages

It is intended in the future that guests are able to send IDM messages to each other along the IDM bus. We should document (and obviously write code to enforce) a security policy that constrains the types of IDM messages guests may send to each other.

feature: implement helper domains for untrusted krata operations

Presently, krata uses scratch space on the dom0 environment to do things like unpack and repack images. We should migrate these operations to dedicated helper domains to derisk them.

The actual design behind how this will work requires further discussion.

chore: convert OCI operations to green threads

Presently OCI operations are not green threads, we should convert them to be green threads for higher performance during OCI operations.

bug: tokio process spawning giving no such process errors inside guest

% ./hack/debug/kratactl.sh exec debian-test sh -c 'apt-get update; apt-get install -y git'
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version (1:2.39.2-1.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Error: exec failed: No child process (os error 10)

The tty support PR at #108 experiences this as well, but it appears tokio Command is causing issues with wait() returning No child process errors.

When investigating this, I would checkout PR #108 as the worktree.

feature: publish prebuilt kernels to GHCR

To simplify user deployment of krata, we should prebuild the kernel and host it on GHCR.

feature: guest attestations

It can be helpful for the guest to provide an attestation back to the control plane via IDM to allow admins to define policies on what configurations / kernels / guestinits are allowed. For example, we could imagine a locked down cluster requiring signed kernel + guestinit, and the guestinit sending an attestation to the control plane to verify that the environment is free of tampering.

VMware appliance compatibility

In order for krata to work with VMware appliances, it would make sense to support OVA files as inputs for guest startup.

feature: forward syslog messages over IDM

In order to capture debug logs from containers, we should have the guestinit bind a unix socket at /dev/log, where it will receive syslog messages, and relay those over the IDM for collection by kratad.

Syslog RFC is https://datatracker.ietf.org/doc/html/rfc5424.

For programs which are linked against libsystemd, we should also create the prerequisite journal configuration so that journal entries are forwarded to /dev/log as well in the form of syslog messages.

feature: provide feedback when converting large OCI images to squashfs

kratactl launch -a cgr.dev/chainguard/powershell for example can take several minutes on the first run as it imports the image and converts it to squashfs.

It would be nice to provide some feedback for the larger images as they are ingested.

feature: bug report through CLI

We can add tooling to capture precise details that might be difficult for people to collect on their own, and we can standardize the collection to make for easier (perhaps even with some automation) troubleshooting.

This feature should:

• Allow for staging the data we're about to send, so it can be inspected before it's sent (pending user interaction, or two-command process?)
• Give a detailed bill of what is collected and why
• Allow for more or less to be submitted with the report, eg with or without memory dumps

feature: utilize system container caches when available to build guest images

Where possible, it makes sense to allow krata to utilize the standard image caches to populate the krata cache from Docker or Podman. This would make it easier to utilize our tools when the container images are pulled from elsewhere.

docs: getting started guide

Create a Getting Started guide that makes it clear how to start using krata as a user.

bug: issues with squashify tool with complex OCI images

I'm trying to convert this image into a squashfs.

I've come across an issue where kratart-squashify will complain that a file already exists when creating the symlink, for example here:

Error: failed to unpack `/var/tmp/krata/krata-compile-66879995-6340-4376-9ea9-75435305fba1/image/bin`

Caused by:
    File exists (os error 17) when symlinking usr/bin to /var/tmp/krata/krata-compile-66879995-6340-4376-9ea9-75435305fba1/image/bin

The odd thing is that /var/tmp/krata/krata-compile-66879995-6340-4376-9ea9-75435305fba1/image/bin is already a symlink to usr/bin.

I wonder if it would be possible to just overwrite in case of the file already existing?

bug: unable to find required guest file: addons.squashfs

What happened?

It seems that now you need to copy ./target/kernel/addons-x86_64.squashfs into /var/lib/krata/guest/addons.squashfs so krata daemon will detect it. So maybe update the point 7 of the development guide.

Version / Commit

84920a8

Relevant log output

$ ./hack/debug/kratad.sh
    Finished release [optimized] target(s) in 0.72s
    Finished dev [unoptimized + debuginfo] target(s) in 0.75s
Error: unable to find required guest file: addons.squashfs

feature: mTLS support for kratad to enable secure network authentication

kratad is currently secured using Unix socket ACLs. We need to implement support for mTLS to allow kratad to be secured over the network, so that a cluster environment can authenticate.