shuffle/shuffle

Shuffle: A general purpose security automation platform. Our focus is on collaboration and resource sharing.

Shuffle Automation

Shuffle is an automation platform for and by the community, focusing on accessibility for anyone to automate. Security operations is complex, but it doesn't have to be.

Key Features — Community & Support — Documentation — Getting Started — Development

Follow us on Twitter at @shuffleio.

Try it

• Self-hosted: Check out the installation guide
• Cloud: Register at https://shuffler.io/register and get cooking (missing a lot of features)

Please consider sponsoring the project if you want to see more rapid development.

Support

• Discord
• Twitter
• Email
• Open issue
• Shuffler.io

Blogposts

• 1. Introducing Shuffle
• 2. Getting started with Shuffle
• 3. Integrating Shuffle with Virustotal and TheHive
• 4. Real-time executions with TheHive, Cortex and MISP

Documentation

Documentation can be found on https://shuffler.io/docs and is written here: https://github.com/shuffle/shuffle-docs.

Setting up a local development environment

Please follow the steps mentioned here!

Related repositories

• OpenAPI apps: https://github.com/shuffle/security-openapis
• Documentation: https://github.com/shuffle/shuffle-docs
• Workflows: https://github.com/shuffle/shuffle-workflows
• Python apps: https://github.com/shuffle/shuffle-apps

Features

• Simple, feature rich workflow editor
• App creator using OpenAPI
• Premade apps for your security tools
• Organization and sub-organization control
• Hybrid resource sharing with shuffler.io (optional)

Website

https://shuffler.io

Contributing

We want to make the world of cybersecurity more accessible and need all the help we can get. Send an email to support@shuffler and we'll make sure to give you any training you may need.

These are the main areas to contribute in:

• Frontend (ReactJS)
• Backend (Golang)
• App Creation (Python & GUI w/OpenAPI)
• Documentation (Markdown)
• Workflow creation (GUI & Conceptualizing)
• Content Creation (Blogs, videos etc)

Contributing guidelines are outlined here.

Contributors

Shuffle

App magicians

License

All modular information related to Shuffle will be under MIT (anyone can use it for whatever purpose), with Shuffle itself using AGPLv3.

Workflows: MIT Documentation: MIT Shuffle backend: AGPLv3 Apps, specification and App SDK: MIT

Architecture

Repository overview

Below is the folder structure with a short explanation

├── README.md				# What you're reading right now
├── backend					# Contains backend related code.
│   ├── go-app 			# The backend golang webserver
│   └── app_sdk			# The SDK used for apps
├── frontend				# Contains frontend code. ReactJS, Material UI and cytoscape
├── functions				# Has execution and extension resources, such as the Wazuh integration
│   ├── onprem				# Code for onprem solutions
│   │   ├── Orborus 	# Distributes execution locations
│   │   ├── Worker		# Runs a workflow
└ docker-compose.yml 	# Used for deployments

Get in touch, send a mail to [email protected] or poke me on twitter @frikkylikeme

shuffle's Issues

Replace NGINX with Caddy

The frontend is currently deployed via NGINX. I would suggest to use (Caddy)[https://caddyserver.com/] instead.

While there is nothing wrong with NGINX, the usage of Caddy has several advantages, for example the automated generation of self signed certificates and out-of-the-box support for Let's encrypt.

I provide the changes as a pull request if you want..

Document process to create and install custom apps

It would be really helpful to document where and how to install custom apps based on the walkoff spec.

Abstract serverless functions to allow multi cloud

The ability to run in AWS or Azure for the serverless functions would be very useful, especially when dealing with stuff that needs to run in a cleared environment (AWS is the only cloud host certified to run TS/SCI and Azure and AWS can run Secret). It would also be useful for when you're at a shop that only supports one of the other clouds (the shop I'm at is AWS and politics would get in the way of using GCP)

Show workflow executions in the workflow itself

Part of the /workflows view (revised) should be view able form the workflow itself for better debugging and live updates.

Missing License

Hey,

Thamk you for the nice work. What's the license of the Shuffle components?

Regards

Add a default starting node

Because of confusion with Schedules and multiple paths, it might be wise to have default start node (essentially an empty app) that's always there. This means you can branch without a testing node or similar.

Add variables as possible static arguments

When using the static field, you might want to use variables in a string.

Say you have the "greetings" variable. If you want to use that one right now, you have to select the variables field. This should be possible by setting the name to "$greetings" in the static field

This is an update to the app_sdk, which should look through the existing variables and map the correct one. Something to decide on: can we have variables, nodes or triggers with the same name? In the case of yes.. why? And what should be prioritized?

Add an alert system that notifies if it sees a specific string

Say you have automation on lots of things, but want to catch if "1.1.1.1" is found anywhere in your results. This can happen immediately as a new result comes in. Notification could be an email or a notification in shuffle itself or even use your selected ticketing system

When attempting to download apps from frikky github getting an error

when attempting to clone apps within the GUI from https://github.com/frikky/shuffle-apps, I am getting an error:

SYNTAXERROR:
JSON.PARSE:
EXPECTED ',' OR '}'
AFTER PROPERTY
VALUE IN OBJECT AT
LINE 1 COLUMN 36 OF
THE JSON DATA

Thanks for your help again.

Arguments/Parameters within a self-created app cannot be set

I tried creating an App "From Scratch" and within the Action i had a variable within the URL. (see {test2})

When the App is implemented into a Workflow the frontend does not provide a way to set the "path" variable.

When executing the Workflow the application throws an error due to the missing path variable

I tried checking out how/where the app resolves the arguments/parameters, but couldn't really find the correct code snippet. All I could find was the parameter setting here, but I couldn't find the backend code which is responsible for serving the parameters.

Backend server gives error while starting go run command

Hello,
I ran the following command while following the installation guide
go run *.go
and encountered the following error:
2020/05/21 09:20:51 Running INIT process 2020/05/21 09:20:51 DBclient error during init: dialing: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. exit status 1

Error while installing packages on ubuntu server

While running sudo apt -y update && sudo apt -y upgrade && sudo apt install -y google-cloud-sdk-app-engine-python google-cloud-sdk-app-engine-python google-cloud-sdk-datastore-emulator google-cloud-sdk-app-engine-go it gives the following error:

E: Unable to locate package google-cloud-sdk-app-engine-python
E: Unable to locate package google-cloud-sdk-app-engine-python
E: Unable to locate package google-cloud-sdk-datastore-emulator
E: Unable to locate package google-cloud-sdk-app-engine-go

I think some additional repositories need to be put before using this command.

Unable to build the frontend

Hello @frikky, sorry for being the first github issuer, I try to use docker-compose to test the tool but faced the following issue

$ docker-compose up -d

Building frontend
Step 1/19 : FROM node as builder
 ---> eaeb579b2c99
Step 2/19 : RUN mkdir /usr/src/app
 ---> Using cache
 ---> 700f54f8d93b
Step 3/19 : WORKDIR /usr/src/app
 ---> Using cache
 ---> e0444ab38af8
Step 4/19 : ENV PATH /usr/src/app/node_modules/.bin:$PATH
 ---> Using cache
 ---> 9480bda5f051
Step 5/19 : COPY package.json /usr/src/app/package.json
 ---> Using cache
 ---> 340d7d88d50f
Step 6/19 : RUN npm install --verbose
 ---> Using cache
 ---> 0213ae6cc070
Step 7/19 : COPY . /usr/src/app
 ---> Using cache
 ---> bc8c2330bd62
Step 8/19 : RUN npm run-script build
 ---> Running in 5a0d49d94431

> [email protected] build /usr/src/app
> react-scripts build

Creating an optimized production build...
Failed to compile.

./src/EditWebhook.js
Cannot find file './assets/img/webhook.png' in './src'.


npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `react-scripts build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-05-12T07_22_20_881Z-debug.log
ERROR: Service 'frontend' failed to build: The command '/bin/sh -c npm run-script build' returned a non-zero code: 1

Please let me know if you need more context.

Cheers

error in userdata on creating admin user

In trying to create an admin user, it tosses "error in user data" on submit.

Create API documentation

API documentation is currently completely lacking. This should be done with OpenAPI, so that Shuffle can use Shuffle without custom builds.

Error with quick deploy

When I am doing docker-compose up with the simple installation.

shuffle-orborus | 2020/05/21 11:27:09 Failed making request: Get "http://shuffle-backend:5010/api/v1/workflows/queue": dial tcp 172.19.0.4:5010: connect: connection refused
shuffle-orborus | 2020/05/21 11:27:12 Failed making request: Get "http://shuffle-backend:5010/api/v1/workflows/queue": dial tcp 172.19.0.4:5010: connect: connection refused
shuffle-orborus | 2020/05/21 11:27:15 Failed making request: Get "http://shuffle-backend:5010/api/v1/workflows/queue": dial tcp 172.19.0.4:5010: connect: connection refused
shuffle-orborus | 2020/05/21 11:27:18 Failed making request: Get "http://shuffle-backend:5010/api/v1/workflows/queue": dial tcp 172.19.0.4:5010: connect: connection refused

Finish general usage documentation

Documentation is here: https://github.com/frikky/shuffle-docs, generated on https://shuffler.io/docs

This includes, but is not limited to:

• Apps (creation, openapi, manual, imports etc)
• Workflows (everything about usability)
• Conditions
• Triggers
• Getting started (important one :))
• Environments
• Users

Frontend: Argument field is not updated when switching between two nodes of the same type/action

Description

I noticed that the argument field does not get updated if the user switches between to nodes that have the same type and action. It seems that the field is also not cleaned correctly

Steps to reproduce:

• Create a testing node with the action "repeat back to me", name it testnode1, set the argument field to "test from testnode1"
• Create a second testing node with the same action ("repeat back to me"), name it testnode2, set the argument to "test from testnode2"
• Select testnode 1 again, notice that the text in arguments won't get updated.
• Add a third testing node, switch the action to "repeat back to me", notice that the argument field is already filled with the value from the previous selected node"

Expected behaviour

The arguments field should display/contain the actual value from the currently selected node. If a new node is created, the argument field should be blank.

Shuffle-database failing to build

Describe the bug
I am looking to test some functionalities of Shuffle and I went to setup the docker environment on my OSX machine and have been running into issues getting Shuffle up

OSX: 10.15.5
Docker Engine:19.03.8
Compose: 1.25.5

I have added the proper mount places to docker under File Sharing(since it is an OSX host).

I am continuing to work through the issue, but figured since this is new and getting under way I would communicate

docker-compose up
Creating network "shuffle_shuffle" with driver "bridge"
Creating shuffle-orborus  ... error
Creating shuffle-database ...

Creating shuffle-database ... error

ERROR: for shuffle-database  Cannot start service database: Mounts denied: EOF

ERROR: for orborus  Cannot start service orborus: Mounts denied: EOF

ERROR: for database  Cannot start service database: Mounts denied: EOF
ERROR: Encountered errors while bringing up the project.
st-rnolen1:Shuffle rnolen$ docker-compose up
Starting shuffle-database ...
Starting shuffle-orborus  ...

ERROR: for shuffle-orborus  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for shuffle-database  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for orborus  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for database  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Debug

docker logs shuffle-backend
Error: No such container: shuffle-backend


docker ps -a
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                         PORTS                                                                                   NAMES
426940011c91        frikky/shuffle:database        "gcloud beta emulato…"   16 minutes ago      Created                                                                                                                shuffle-database
504f685af6bb        frikky/shuffle:orborus         "./orborus"              16 minutes ago      Created                                                                                                                shuffle-orborus

Backend search for openapi's out there

This should be a part of the /apps view, where if there are no other apps, search the backend for OpenAPI, and then make it possible to import them.

Create dashboard with statistics

A generic dashboard is available at /dashboard and has stats about executions, apps etc. There is however no dashboard.

CORS misconfiguration for API server

The webapp backend is most likely misshandling CORS requests.
This code snippet sets the Access-Control-Allow-Origin response Header to whatever value the Origin Request header is set.

This behaviour allows for potential CORS-based attacks.
Instead of simply reflecting the request Origin header, a CORS whitelist with trusted sites should be created.

Example Request (The Header value: attacker-controlled-value is reflected within the response)

GET /api/v1/apps HTTP/1.1
Host: localhost:3443
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: application/json
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json
Connection: close
Origin: attacker-controlled-value

Create UI for getting data from a previous node in static text field

Currently, the only way to get data from a previous node is to write out e.g.:
$node_name

This should be possible with the click of a button, allowing you to choose the value you want.

On top of this, if the node (or action in general) has been executed successfully previous, that data should be view able, so that you potentially can choose the JSON value (or other data form). This is dynamic data loading / predictive data choice of sorts, rather than pre-defining these in the apps themselves. Both together might be a good way to go.

Create generic app categories

The point of this is to make generic workflows non-problematic.
E.g. you have "Tickets" or "Alerts", and when you authenticate e.g. "TheHive" or "Service now".

Same for "SIEM" with Qradar, ELK, Splunk etc.

More plug and play.

This leads into detection and prevention out of the box.

Add execution variable to handle basic multi-tenancy

Idea came from a talk with an MSSP. Point is to have a way to use different auth config with the same workflow.

Another way: have auth be its own specific thing that's described when executing every workflow. This could be another feature like environment.

webapp binary in the "backend" directory

Within the "backend->app" directory here the compiled webapp is present. I'm not sure if this is intentional, or this was an oversight.
If its intentional it would probably a good idea to compile the webapp during the Docker build process.

Also within the directory is a readme which probably belongs to the "backend" database development setup, as it "describes" the gcloud database setup.

Add override variable for Dockerfile for app creation

This is related to #57 and that the backend already has a Dockerfile installed under app_gen.

Make triggers able to start on any action

The "starting_node" value can be modified, as per the "User Input" continuation schedule. This should make it trivial to make the schedule, webhook etc. able to connect anywhere.

Docker app sdk and worker exits unexpectedly

Hello,
I tried starting shuffle with the docker command and the images stop unexpectedly because of which I m getting 502 error while doing the admin setup it seems.

Create general workflows based on Mitre Att&ck

This would most likely be based on #31, and is a bit further in the future

Create encrypted and global variables

Variables are currently only availble per workflow. Make global ones and make them safe for distribution

Consider the possibility of using Kubernetes

Docker compose is good for portability, however with Kubernetes able to run on windows through docker desktop, adding kubernetes support and creating helm charts would help enable more flexibility in cloud deployment, as well as open new doors like the ability to use prometheus for monitoring.

Think server is down for docs

I see 500 Internal Server Error when move to this page: https://shuffler.io/docs/about

Screenshot:

Make versioned Docker images

Instead of frikky/shuffle:frontend
Do shuffle/frontend:1.0.0

Or similar

Problems creating apps with SSL certificate

Hi Frederik, i'm trying to create a QRadar app but I can't connect to my server because it has an SSL certificate and when i try, it gives me Authentication Error because of SSL. Is it posible to create a box to enable/disable SSL verification or a way to introduce my SSL file? I've tried to enter -k argument into curl request when I create actions but it doesn't accept it.

Thank you very much

.env not respecting docker-compose.yml

Describe the bug
At MacOS the docker is not starting because of:
ERROR: for database Cannot start service database: Mounts denied:
The path /etc/shuffle
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.

Try to change File Sharing preferences as described in docker-compose.yml, but still got the error. Finally updating the .env file to the same location works.

Create default workflows that loads on init

There should exist default workflows that are loaded for the user at some point. These should reflect the default apps.

Repository: https://github.com/frikky/shuffle-workflows

Be able to import and share workflows

Export is available, so why not import?
What's necessary for this? It's quite easy to do directly, but checking apps seems important.

Add proxy config option

Re: Chat with @lucyel on gitter - There might be issues with proxies that aren't sorted with just Docker if Shuffle doesn't look for the proxy options. Should be defined in .env.

Service 'orborus' failed to build: The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o orborus .' returned a non-zero code: 2

Building orborus
Step 1/9 : from golang as builder
 ---> 7e5e8028e8ec
Step 2/9 : RUN mkdir /app
 ---> Using cache
 ---> 9cf892e7bd5c
Step 3/9 : WORKDIR /app
 ---> Using cache
 ---> e76e061c9822
Step 4/9 : COPY orborus.go /app/orborus.go
 ---> Using cache
 ---> 1db44e43f4ed
Step 5/9 : RUN go get github.com/docker/docker/api/types github.com/docker/docker/api/types/container github.com/docker/docker/client
 ---> Using cache
 ---> 2fb9da799ac8
Step 6/9 : RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o orborus .
 ---> Running in 4fc5643ab6fd
# _/app
./orborus.go:74:34: not enough arguments in call to cli.ContainerCreate
	have (context.Context, *container.Config, *container.HostConfig, nil, string)
	want (context.Context, *container.Config, *container.HostConfig, *network.NetworkingConfig, *v1.Platform, string)
ERROR: Service 'orborus' failed to build: The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o orborus .' returned a non-zero code: 2

I get this error while composing the docker.

Workflows stuck on "EXECUTING" status

I've been installing and trying to play with the app these past hours and all my workflows (on the "on prem" version from your git) are stuck with : Status: EXECUTING

The same workflows are working on the cloud-based version, and (when used) my webhooks are functionning (and returning correct data when posting).

My last try was a basic "Hello World" workflow, which is also stuck.
Is there anything I missed that I'll need to configure ?

I also have some questions about the "Environments" and the possible multitenancy, do you prefer me to send you a mail about these ?

Anyway, thank you very much because this project seems really cool and I'll try to put some effort on it :) !

Make for-loops controllable

For-loops are currently possible with e.g. $exec.listname.#.name, grabbing ALL "names" from the execution argument.

This should be a fix in the app_sdk. It also requires a frontend of sorts.

Hello world not finishing

As requested, in https://medium.com/security-operation-capybara/getting-started-with-shuffle-an-open-source-soar-platform-part-2-1d7c67a64244

I've got Centos8 in a VM, all working up to the point of running the test app with hello world. It says Executing Workflow, status is executing, started timestamp looks ok and the ID is a nice long UUID type string, the actions section just has spinning red circles.

Add certficate control and guidance

https is vital. As it uses a nginx configuration, this isn't too hard

Frontend: Check if unsaved changes exist before starting the workflow

When starting a workflow, the frontend does currently not check if there are any unsaved changes. This might cause irritations, as the general expectation is that the currently displayed workflow gets executed.

Suggestion:
When pressing the play button, the frontend must check if the current workflow contains unsaved changes. In this case, the user should be asked if he wants to save the workflow before continuing, similar to a IDE...

It might also be great to have some kind of "auto-save" property, bound to the user and/or project.

Check if App is in use before allowing App deletion

It is currently possible to delete an existing app, even if it is used within a workflow.
It could be a nice usability feature to check if the app is currently in use, before a user is allowed to delete it. This would make it harder for a user to silently break existing workflows.

Add app authentication control

Currently, apps are controlled per workflow. Make it possible to authenticate, and reuse it

Workflow execution issue - webhooks

shuffle_app_sdk_1 exited with code 0
worker_1    | 2020/05/22 15:53:49 Setting up worker environment
worker_1    | 2020/05/22 15:53:49 No AUTHORIZATION key set in env
worker_1    | panic: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
worker_1    | 
worker_1    | goroutine 1 [running]:
worker_1    | main.shutdown(0x0, 0x0, 0x0, 0x0)
worker_1    | 	/home/frikky/git/shuffle/functions/onprem/worker/worker.go:229 +0x69e
worker_1    | main.main()
worker_1    | 	/home/frikky/git/shuffle/functions/onprem/worker/worker.go:788 +0xc99
shuffle_worker_1 exited with code 2

I have been getting this error. Also I guess it has something to do with reason on why my test workflow is also not working.

Can you let me know whats the issue going on here ?

Default user name password are missing in installation guide.

Doubt this is a bug but error when running "docker-compose up -d"

After cloning from Github I ran the docker-compose command and I am getting a certificate error while attempting to build backend:

go: cloud.google.com/[email protected]: Get "https://proxy.golang.org/cloud.google.com/go/@v/v0.57.0.mod": x509: certificate signed by unknown authority
ERROR: Service 'backend' failed to build: The command '/bin/sh -c go get -v' returned a non-zero code: 1

I have updated ca-certiifcates and no luck, is there a location in you image where I can excluded SSL validation for go?