expediagroup/flyte

Flyte binds together the tools you use into easily defined, automated workflows

Table of contents

• Overview
• Getting Started
• Running
• Running the tests
• More resources
• Contributing
• License

Overview

Flyte binds together the tools you use into easily defined, automated workflows. It is a request-based, decoupled automation engine which allows you to define flows to enable integrated behaviour across these disparate tools.

Flyte has chat-ops enabling integrations for Slack, as well as some other out-of-the-box integrations. These integrations, or packs can be added to and extended easily by using Flyte's RESTful API.

Some of the applications already in use include chat-ops based inventory management, host/container administration and orchestration, and deployment of applications into Kubernetes.

Automation is done using flows which essentially take form of "if this happens in system A, then do this in system B". For example you could create a flow that triggers sending an email to a team's email if their app's deployment has failed, or create a deployment pipeline triggered from an instant chat message.

How it works

The Flyte-API acts as the orchestrator and is backed by a (MongoDB) database server in which Flows, Packs references and Datastore entries are stored. Before continuing with the high level architecture diagram, lets introduce some of the key concepts of Flyte:

• Flows are a list of steps that define a particular use case - for example triggering the deploy of an app when a user types the message "deploy foo-app 1.2.0" in a particular chat room. Each step in a flow consists of:

  • An Event that triggers it (e.g. an instant message being observed in a particular chat room).
  • A Criteria that must be satisfied for the step to run (e.g: message matches certain regex).
  • An Action to be executed if the criteria matches.

  You can find more info about flows here.

• Packs are self-contained apps that are responsible for executing flow actions and sending events to the flyte-api. For instance, flyte-slack-pack consumes events/messages from a slack channel but also sends messages via Slack.

• DataStores are basically configuration properties shared between flows. For instance, list of environments, urls, etc.

How are flows executed

1. Packs consume events from external services - for example, a Slack message from a specific channel.
2. Packs transform these external events to flyte events before pushing them to Flyte-API.
3. Flyte-API receives an event and triggers any flow which is listening to that event.
4. This flow execution will create an action and flyte-api will store it in its database.
5. Packs will poll for new actions to Flyte-API.
6. Flyte-API will assign an action to a pack.
7. Pack will execute the action and return the result as a new event, which will trigger the next step in our current flow or a different one.

Flyte-API / Flyte Pack interaction: The Flyte Pack queries Flyte-API which returns a response.

Getting Started

Check out the Quick Start documentation to get started on building new flows or custom packs.

Running

The are a number of ways to run flyte and its mongo db. Note that the default mongo host and port for flyte is localhost:27017 (this value can be changed using the FLYTE_MGO_HOST env variable).

The port number can be overridden, see Configuration

Once running, flyte will be available on http://localhost:8080 (TLS disabled), or http://localhost:8443 (TLS enabled).

From Source

Pre-req: must have modules installed and Go 1.11 or higher.

Build & run:

make run

Command starts mongo docker container and flyte go executable in the background. Output is written to flyte.out file.

or manually...

go test ./... -tags="integration acceptance" //remove tags if only want to run unit tests
docker run -d -p 27017:27017 --name mongo mongo:latest
go build && ./flyte

Using Docker

make docker-run

or manually...

docker run -d -p 27017:27017 --name mongo mongo:latest
docker build -t flyte:latest .
docker run -p 8080:8080 -e FLYTE_MGO_HOST=mongo -d --name flyte --link mongo:mongo flyte:latest
 

Running the tests

Acceptance Tests

For acceptance tests:

go test ./... -tags=acceptance

These will start a disposable docker mongo container and flyte on randomly available TCP ports. If mongo can't be started (e.g docker is not available in the path), tests will be skipped and won't fail the build.

Acceptance tests will not run when building flyte in docker.

The tests can be run in an IDE by running the test suite in "acceptance_test.go".

Integration Tests

For integration tests:

go test ./... -tags=integration

For the mongo db integration tests, which are slower running:

go test ./... -tags=db

To run both:

go test ./... -tags="integration db"

Please note that both unit and integration tests will run using the above command/s.

Postman

There are a number of postman files in postman folder that can be used to test running flyte

More resources

• Security
• Audit
• Configuration
• Flows
• Packs
• DataStores

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

License

This project is licensed under the Apache License License - see the LICENSE file for details

flyte's Issues

Flyte API swagger docs links are always using the same protocol and host

Flyte API swagger docs links are always the same regardless of the request protocol and host of X-Forwarded... Headers this is because map for docs URIs is global and is created once on the first request and whatever protocol and host was used will be used for lifetime of the instance.
related file: flytepath/uridocs.go

Pack registration accepting HATEOAS links fields (and others)

Pack registration accepts links, lastSeen and status fields as part of the pack input data. This data shouldn't be part of the input data but part of the response.

https://github.com/HotelsDotCom/flyte/blob/4b4be70c174ae93cdde8bc3dbc1b0c438bd0780f/pack/pack.go#L33

Embedding pack and adding these fields to packResponse should fix this.

https://github.com/HotelsDotCom/flyte/blob/4b4be70c174ae93cdde8bc3dbc1b0c438bd0780f/pack/view.go#L25

Someone could potentially add custom links to our "HATEOAS response" and clients could call an unexpected endpoint.

Investigate issue with logging

When investigating a recent issue, the logging that was being emitted from the bamboo pack appears to all have been on line 58 of whatever file outputted it, even when no code was on that line …

makefile run executes in background but output is in current shell

would be nice to log in a file and detach it as well (so when user closes terminal, flyte is still running and logs can be viewed)

Misuse of Request.Proto field

Request interceptor in httputil/requestinterceptor.go is misusing Proto field. This field is ment to be used for:

// The protocol version for incoming server requests.
//
// For client requests these fields are ignored. The HTTP
// client code always uses either HTTP/1.1 or HTTP/2.
// See the docs on Transport for details.
Proto string // "HTTP/1.0"

We are setting it to http or https which will be used in other places to construct URLs while leaving Request.URL field unmodified. Above is snippet from the documentation and example value for Proto field

Include example flow(s)

Including example flow(s) would help people understand how Flyte operates & how to create flows.

Differentiate between different Flytes

Thanks for making this project open source!

Is there any interest in clearly differentiating this repo from https://github.com/flyteorg/flyte in the readme or other prominent location? This could reduce some confusion. I'm happy to put a PR for this but I first wanted to check if it would be considered.