michaelwasher/kstrace

A kubectl plugin used to collect strace data from running Pods

kstrace

This tool is a kubectl plugin that is used to collect strace data from Pods running with your Kubernetes cluster.

Description

This tool is a kubectl plugin used for watching and reviewing system-calls from processes running inside Pods in a Kubernetes cluster.

The tool creates a privileged Pod in the cluster which will 'attach' to the running target Pod and will stream the strace data back to the end user.

This application also allows for strace monitoring of multiple Pods (DaemonSets, Deployments, Services) at the same time by streaming the results back into a designated folder.

Installation

Install using Krew:

kubectl krew install strace

Build from source:

# Build for the current OS
make build && cd bin/

# Cross-compile for all supported OSes
make all && cd bin/

Getting Started

To start watching a Pods strace calls on the command-line:

kubectl strace -o - <pod>

Multiple Pods or containers can be traced at the same time and collected into folders.

kubectl strace --trace-timeout=30s deployment/<deployment>

The kstrace application can trace the following Kubernetes resources identified by either their long name or short name: Deployment, DaemonSet, Service, Pod

The command flags for kstrace are listed below:

      --image string             The trace image for use when performing the strace. (default "quay.io/mwasher/crictl:0.0.2")
      --log-level string         The verbosity level of the output from the command. Available options are [panic, fatal, error, warning, info, debug, trace]. (default "info")
  -n, --namespace string         If present, the namespace scope for this CLI request
  -o, --output string            The directory to store the strace data. (default "strace-collection")
      --socket-path string       The location of the CRI socket on the host machine. (default "/run/crio/crio.sock")
      --trace-timeout string     The length of time to capture the strace output for. (default "0")

Limitations

Auto-completion of Kubectl plugins is currently not possible but is an active development. Kubernetes Issue 74178

When this functionality is merged into the Kubectl tool, this tool will be updated to be compliant with the Kubectl completion mechanisms.

Credits

This tool was inspired heavily by Ksniff. The ksniff tool provides a similar functionality for collecting network packet captures (PCAP) and was created by Eldad Rudich and maintained by Robert Bost. Although this tool was written independently to the KSniff tool, the developer for this tool has also worked on the Ksniff tool and portions may resemble the design of KSniff.

kstrace's Issues

Created namespace set pod admission policy

If my cluster has a pod admission policy default of restricted the created namespace doesn't permit attaching in the way this intends.

Can the correct annotation be attached to the generated namespace. https://kubernetes.io/docs/concepts/security/pod-security-standards/

Write Krew Plugin YAML definitions

Write Krew Plugin YAML definitions and publish the first version

Write a e2e smoke test for all resource types and one for each flag option

Write an e2e test for all supported resource types: [Deployment, DaemonSet, Service, Pod]
Write e2e test for --output, --trace-timeout, --socket-path, --namespace, --image, --log-level

Add Tracing for Multiple Pods at a time

Add the functionality to allow tracing for multiple Pods at the same time

Close Gracefully

Capture Signals and close gracefully by removing all of the created resources

Limit Tracing to one tracePod per node

Currently if there are multiple target pods on the same node, each target Pod will have a tracePod created. The same could be achieved with a single tracePod created and running multiple exec commands (SDPY connections) to trace multiple PIDs.

This should not change the functionality but will just be an increase in performance.

Add tracing functionality to sniff for all matching to a Service, DaemonSet and Deployment

Blocker: #8

Add the ability to run the following commands and strace data be collected for all of the resulting Pods:

kubectl strace ds/<daemonset>
kubectl strace deploy/<daemonset>
kubectl strace svc/<svc>

Do not interlace the results but instead collect the output into a folder for each Pod and a file for each container trace.

containerd support

does it will support containerd?

Infer container runtime socket

Infer the container runtime socket location from the containerID of the target Pod and the defaults for each runtime. Currently just allows for overwrite

Split tracer pod creation into a generic library for privileged attaching to Pods

Clean up imports and modules

Clean up go modules and re-order imports to group relevant imports

Automate version releases and update code tagging on release

There are a number of places within the code that reference the release version. This needs to be replaced with a value that is hard-coded at build-time and is associated with the Github Tag on the workflow.

This needs to be set in at least the Krew version and the kubectl strace version command. This should also push up a docker-image with the matching version of crictl on Quay and update the default --image value.