snowflake-labs/sansshell

A non-interactive daemon for host management

SansShell

A non-interactive daemon for host management

flowchart LR;

subgraph sanssh ["sansshell client (sanssh)"]
    cli;
    client;
    subgraph client modules
      package([package]);
      file([file]);
      exec([exec]);
    end
    cli --> package --> client;
    cli --> file --> client;
    cli --> exec --> client;
end
subgraph proxy ["proxy (optional)"]
    proxy_server[proxy-server];
    opa_policy[(opa policy)];
    proxy_server --> opa_policy --> proxy_server
end
subgraph sansshell server ["sansshell server (on each host)"]
    server[sansshell-server];
    host_apis;
    s_opa_policy[(opa policy)];
    subgraph service modules
      s_package([package]);
      s_file([file]);
      s_exec([exec]);
    end
    server --> s_package --> host_apis;
    server --> s_file --> host_apis;
    server --> s_exec --> host_apis;
    server --> s_opa_policy --> server
end
user{user};
user --> cli;
client --"gRPC (mTLS)"--> proxy_server
proxy_server --"grpc (mTLS)"---> server

SansShell is primarily a gRPC server with a variety of options for localhost debugging and management. Its goal is to replace the need to use an interactive shell for emergency debugging and recovery with a much safer interface. Each authorized action can be evaluated against an OPA policy, audited in advance or after the fact, and is ideally deterministic (for a given state of the local machine).

sanssh is a simple CLI with a friendly API for dumping debugging state and interacting with a remote machine. It also includes a set of convenient but perhaps-less-friendly subcommands to address the raw SansShell API endpoints.

Getting Started

How to set up, build and run locally for testing. All commands are relative to the project root directory.

Building SansShell requires a recent version of Go (check the go.mod file for the current version).

Build and run

You need to populate ~/.sansshell with certificates before running.

$ cp -r auth/mtls/testdata ~/.sansshell

Then you can build and run the server, in separate terminal windows:

$ go run ./cmd/sansshell-server
$ go run ./cmd/sanssh --targets=localhost file read /etc/hosts

You can also run the proxy to try the full flow:

$ go run ./cmd/sansshell-server
$ go run ./cmd/proxy-server
$ go run ./cmd/sanssh --proxy=localhost:50043 --targets=localhost:50042 file read /etc/hosts

Minimal debugging UIs are available at http://localhost:50044 for the server and http://localhost:50046 for the proxy by default.

Environment setup : protoc

When making any change to the protocol buffers, you'll also need the protocol buffer compiler (protoc) (version 3 or above) as well as the protoc plugins for Go and Go-GRPC

On MacOS, the protocol buffer can be installed via homebrew using

brew install protobuf

On Linux, protoc can be installed using either the OS package manager, or by directly installing a release version from the protocol buffers github

Environment setup : protoc plugins

On any platform, once protoc has been installed, you can install the required code generation plugins using go install.

$ go install google.golang.org/protobuf/cmd/protoc-gen-go
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
$ go install github.com/Snowflake-Labs/sansshell/proxy/protoc-gen-go-grpcproxy

Note that, you'll need to make certain that your PATH includes the gobinary directory (either the value of $GOBIN, or, if unset, $HOME/go/bin)

The tools.go file contains helpful go generate directives which will do this for you, as well as re-generating the service proto files.

$ go generate tools.go

Creating your own certificates

As an alternative to copying auth/mtls/testdata, you can create your own example mTLS certs. See the mtls testdata readme for steps.

Debugging

Reflection is included in the RPC servers (proxy and sansshell-server) allowing for the use of grpc_cli.

If you are using the certificates from above in ~/.sansshell invoking grpc_cli requires some additional flags for local testing:

$ GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=$HOME/.sansshell/root.pem grpc_cli \
  --ssl_client_key=$HOME/.sansshell/client.key --ssl_client_cert=$HOME/.sansshell/client.pem \
  --ssl_target=127.0.0.1 --channel_creds_type=ssl ls 127.0.0.1:50043

NOTE: This connects to the proxy. Change to 50042 if you want to connect to the sansshell-server.

A tour of the codebase

SansShell is composed of 5 primary concepts:

1. A series of services, which live in the services/ directory.
2. A server which wraps these services into a local host agent.
3. A proxy server which can be used as an entry point to processing sansshell RPCs by validating policy and then doing fanout to 1..N actual sansshell servers. This can be done as a one to many RPC where a single incoming RPC is replicated to N backend hosts in one RPC call.
4. A reference server binary, which includes all of the services.
5. A CLI, which serves as the reference implementation of how to use the services via the agent.

Services

Services implement at least one gRPC API endpoint, and expose it by calling RegisterSansShellService from init(). The goal is to allow custom implementations of the SansShell Server to easily import services they wish to use, and have zero overhead or risk from services they do not import at compile time.

List of available Services

1. Ansible: Run a local ansible playbook and return output
2. Execute: Execute a command
3. HealthCheck
4. File operations: Read, Write, Stat, Sum, rm/rmdir, chmod/chown/chgrp and immutable operations (if OS supported).
5. Package operations: Install, Upgrade, List, Repolist
6. Process operations: List, Get stacks (native or Java), Get dumps (core or Java heap)
7. MPA operations: Multi party authorization for commands
8. Service operations: List, Status, Start/stop/restart

TODO: Document service/.../client expectations.

The Server class

Most of the logic of instantiating a local SansShell server lives in the server directory. This instantiates a gRPC server, registers the imported services with that server, and constraints them with the supplied OPA policy.

The reference Proxy Server binary

There is a reference implementation of a SansShell Proxy Server in cmd/proxy-server, which should be suitable as-written for many use cases. It's intentionally kept relatively short, so that it can be copied to another repository and customized by adjusting only the imported services.

The reference Server binary

There is a reference implementation of a SansShell Server in cmd/sansshell-server, which should be suitable as-written for some use cases. It's intentionally kept relatively short, so that it can be copied to another repository and customized by adjusting only the imported services.

The reference CLI client

There is a reference implementation of a SansShell CLI Client in cmd/sanssh. It provides raw access to each gRPC endpoint, as well as a way to implement "convenience" commands which chain together a series of actions.

It also demonstrates how to set up command line completion. To use this, set the appropriate line in your shell configuration.

# In .bashrc
complete -C /path/to/sanssh -o dirnames sanssh
# Or in .zshrc
autoload -Uz compinit && compinit
autoload -U +X bashcompinit && bashcompinit
complete -C /path/to/sanssh -o dirnames sanssh

Multi party authorization

MPA, or multi party authorization, allows guarding sensitive commands behind additional approval. SansShell supports writing authorization policies that only pass when a command is approved by additional entities beyond the caller. See services/mpa/README.md for details on implementation and usage.

To try this out in the reference client, run the following commands in parallel in separate terminals. This will run a server that accepts any command from a proxy and a proxy that allows MPA requests from the "sanssh" user when approved by the "approver" user.

# Start the server
go run ./cmd/sansshell-server -server-cert ./auth/mtls/testdata/leaf.pem -server-key ./auth/mtls/testdata/leaf.key
# Start the proxy
go run ./cmd/proxy-server -client-cert ./services/mpa/testdata/proxy.pem -client-key ./services/mpa/testdata/proxy.key -server-cert ./services/mpa/testdata/proxy.pem -server-key ./services/mpa/testdata/proxy.key
# Run a command gated on MPA
go run ./cmd/sanssh -client-cert ./auth/mtls/testdata/client.pem -client-key ./auth/mtls/testdata/client.key -mpa -proxy localhost -targets localhost exec run /bin/echo hello world
# Approve the command above
go run ./cmd/sanssh -client-cert ./services/mpa/testdata/approver.pem -client-key ./services/mpa/testdata/approver.key -proxy localhost -targets localhost mpa approve 53feec22-5447f403-c0e0a419

Extending SansShell

SansShell is built on a principle of "Don't pay for what you don't use". This is advantageous in both minimizing the resources of SansShell server (binary size, memory footprint, etc) as well as reducing the security risk of running it. To accomplish that, all of the SansShell services are independent modules, which can be optionally included at build time. The reference server and client provide access to the features of all of the built-in modules, and come with exposure to all of their potential bugs and bloat.

As a result, we expect most users of SansShell would want to copy a very minimal set of the code (a handful of lines from the reference client and server), import only the modules they intend to use, and build their own derivative of SansShell with more (or less!) functionality.

That same extensibility makes it easy to add additional functionality by implementing your own module.

To quickly rebuild all binaries you can run:

$ go generate build.go

and they will be placed in a bin directory (which is ignored by git).

TODO: Add example client and server, building in different SansShell modules.

If you need to edit a proto file (to augment an existing service or create a new one) you'll need to generate proto outputs.

$ go generate tools.go

NOTE: tools.go will need to have additions to it if you add new services.

sansshell's Issues

Multi Party Authorization

Within Snowflake, we're encountering some use cases where we'd benefit from the ability to require one or more human approvals to a Sansshell command at a per-command granularity. This issue tracks any changes related to adding that feature.

Add DNS lookup features

It would be useful to be able to perform DNS queries from the perspective of hosts in production running sansshell, in order to debug e.g. private DNS zone issues inside cloud providers, or host DNS configuration issues.

Ideally any implementation would use the system resolver (i.e. libresolv, /etc/resolv.conf etc etc.). I understand the net/dns in Go does this by default, so hopefully will not be too difficult.

Add project to OPA ecosystem listing

Not an issue really, but just found out about this project and I love the idea! 😃

Would you consider adding this integration to the OPA ecosystem page? (there's a link top-right for how to add an entry)

Also, definitely would welcome a PR to have it added to the Awseome OPA list.

Thanks!

localfile unit tests fail on Ubuntu 22.04

Localfile tests fail on ubuntu 22.04

--- FAIL: TestSetFileAttributes (0.00s)
    localfile_test.go:577: nobody gid: err was group: unknown group nobody, want nil
stat: /tmp/TestList3805605923/001 - /tmp/TestList3805605923/001/testfile.3508715379
stat: /tmp/TestList3805605923/001/testfile.3508715379 - /tmp/TestList3805605923/001/testfile.3508715379
FAIL
exit status 1
FAIL	github.com/Snowflake-Labs/sansshell/services/localfile/server	2.194s
s

This is due to nobody's group being called "nogroup". Rather than lookup the group by name, I think finding the group name by going from user to primary group id to group name will be more robust.

Symlink handling in `file` action inappropriately follows symlinks

We're following symlinks when doing actions like listing directories. For example, let's set up a badly-formed directory structure.

% mkdir /tmp/repro
% touch /tmp/repro/a /tmp/repro/c  /tmp/repro/d
% ln -s broken /tmp/repro/c
% ln -s /tmp/repro/d /tmp/repro/b

Running ls normally handles this fine.

% ls -l /tmp/repro
total 0
-rw-r--r--@ 1 srhodes  wheel   0 Apr  3 09:11 a
lrwxr-xr-x@ 1 srhodes  wheel  12 Apr  3 09:13 b -> /tmp/repro/d
lrwxr-xr-x@ 1 srhodes  wheel   6 Apr  3 09:12 c -> broken
-rw-r--r--@ 1 srhodes  wheel   0 Apr  3 09:11 d

SansShell's file action has issues, following the symlink and failing when it can't do so.

% ./sanssh --targets=localhost file ls --long /tmp/repro
-rw-r--r--   -      501        0                0 Mon Apr  3 16:11:57 UTC 2023 /tmp/repro/a
-rw-r--r--   -      501        0                0 Mon Apr  3 16:11:57 UTC 2023 /tmp/repro/b
Got error from target localhost:50042 (0) - rpc error: code = Internal desc = stat: os.Stat error stat /tmp/repro/c: no such file or directory