Giter Site home page Giter Site logo

elsy's Introduction

elsy

Build Status

elsy (also known as lc, which is what the binary is called) is an opinionated, multi-language, build-tool based on Docker and Docker Compose. It allows organizations to implement a consistent build workflow across many different repos, spanning a wide array of programming languages.

elsy will not replace your favorite build tool, it is simply a thin wrapper that:

  • provides a consistent development workflow across all repos using elsy
  • provides the ability to fully test Docker images from a blackbox perspective
  • reduces local-dev tool requirements to the bare minimum, regardless of programming language (i.e., you only need to install Docker, Compose, and elsy)
  • ensures consistent builds regardless of environment (i.e., fixes the "works on my machine" problem since the repo defines its exact dependency requirements)

With elsy, it is possible to build, test, and publish a repo from scratch with just:

git clone <repo>
cd repo
lc ci

Getting Started

Prerequisites: elsy requires both Docker and Docker Compose.

Follow the below steps to install elsy:

## choose platform (darwin or linux) and versions (see releases page)
PLATFORM=darwin
VERSION=1.7.0

## install binary for your system
curl -fL -o /usr/local/bin/lc https://github.com/cisco/elsy/releases/download/v$VERSION/lc-$PLATFORM-amd64-v$VERSION
chmod +x /usr/local/bin/lc

As of version v2.1.0 of Elsy, you can upgrade to the latest verison by running

lc system upgrade

Previous versions require repeating the installation steps to manually download and install the binary.

See the Using elsy in a Project document for info on how to setup a repo to use elsy.

The Lifecycle

At its core, elsy is an implementation of a build lifecycle that generalizes to any repo producing a software artifact. Running lc ci will execute the full build lifecycle inside a repo, it is made up of the stages defined in the following sub-sections. lc ci operates in a fail-fast mode, so if any stage fails, the following stage will not be run.

See the examples folder for concrete examples of this lifecycle in action.

See elsy Best Practices for guidance on how to use elsy for typical development workflows.

lc teardown

Running lc teardown simply tells elsy to clean up any state that might be left over from a previous build.

lc bootstrap

Running lc bootstrap will setup a new repo and make sure all dependencies (e.g., docker images, external software libs) are downloaded and built. Thanks to Docker caching, this step is only time-intensive the first time it is run.

If present, bootstrap will call the repo's docker-compose installdependencies service that will execute repo-specific command(s) to install external libraries. See the docker-compose.yml file inside the elsy repo itself for an example of this.

lc clean

Running lc clean will ensure artifacts from previous builds are removed. Typically, this service is used before starting a new build. This is analogous to running mvn clean before running mvn package, for example. The mvn, lein, make, and sbt templates all define clean services, so if the project uses one of those, no additional work is required.

The difference between clean and teardown, which both perform similar actions, is that teardown only disposes of containers, whereas clean can remove artifacts from the local disk.

lc test

Running lc test will execute the repo's docker-compose test service, which will execute repo-specific command(s) to run all unit and integration tests for the code in that repo.

lc package

Running lc package will do two things. First it will execute the repo's (optional) docker-compose package service, which will execute repo-specific command(s) for packaging the repo's code into the final artifact. Second, if a Dockerfile is found in the root of the repo, elsy will build that Dockerfile into a new Docker image that is ready for final testing and publishing.

Note, when run on its own, lc package will also run lc test to ensure you are packaging working code, you can prevent this by using the --skip-tests flag.

When using elsy with Docker 1.11.1 and higher, lc package will apply the following image labels during build time:

  • com.elsy.metadata.git-commit=<git-commit> - The git commit that the image was built from. The value of <git-commit> is taken from the GIT_COMMIT env var (it is up to your build system to populate this env var).

lc blackbox-test

This is where the real power of docker-based development comes into play.

Running lc blackbox-test will execute the repo's docker-compose blackbox-test service to run repo-specifc logic for testing the final artifact of the repo. This means that it is possible to test the real container before releasing it to production.

For example, if the repo is producing a Docker-based microservice that uses a Mysql database, the blackbox-test service will:

  1. stand up the microservice container that was just packaged during lc package
  2. stand up a mysql container (and initialize the schema) for the microservice to use
  3. execute API-level tests against the microservice container to ensure it is functioning correctly with the database

Note, when run on its own, lc blackbox-test will also run lc package to ensure you are testing the latest code, you can prevent this by using the --skip-package flag.

You can also run the blackbox tests by running lc bbtest.

At the end of the blackbox-test run, regardless of the outcome, all associated containers will be torn down. If you wish to leave them up, pass the --keep-containers option.

lc publish

Running lc publish does two things: First it will execute the repo's (optional) docker-compose publish service that will run repo-specific command(s) for publishing an artifact. This custom service is typically used for repos that do not produce Docker images.

Second, if a Docker image was created during the lc package phase, elsy will correctly tag and publish that image to the registry defined in the lc.yml file.

lc publish uses the following rules when deciding what to publish:

For running the custom publish service:

  • elsy will only run the custom publish service on branches with the pattern of: origin/master or origin/release/<name>, or on a valid elsy release git tag.

For tagging Docker images:

  • If the git branch is origin/master, elsy will apply the tag latest to the docker image.
  • If the git branch is origin/release/<name> elsy will apply the tag <name>
  • If the git branch is origin/feature/<name>, elsy will apply the tag snapshot.feature.<name> to the docker image.
  • If the git branch is origin/<name>, elsy will apply the tag snapshot.<name>
  • If a git tag exists and it is a valid elsy release tag, elsy will use that tag as the docker image tag.

If you have defined a custom publish service in your docker-compose.yml, elsy will pass the service an env var called LC_PUBLISH_DOCKER_TAG that contains the tag elsy will use for the docker image, you just need to delcare the env var like so:

publish:
  image: busybox
  environment:
    - LC_PUBLISH_DOCKER_TAG
  command: echo custom publish of tag $LC_PUBLISH_DOCKER_TAG

Valid Git Relase Tag:

elsy currently considers a valid git release tag to be any tag following the schema:

vX.Y.Z[-Q]

Where X, Y and Z are integers representing the Major, Minor, and Patch version (respectively) and Q is an optional string qualifier. In the future we plan to make this schema configurable.

lc run

Running lc run will run a specific service that is contained in the docker-compose.yml file. This is equivalent to lc dc run ....

elsy Templates

The elsy lifecycle manifests itself in subtly different ways depending on the underlying build tool. elsy ships with a small set of pre-baked templates (e.g., mvn, sbt) that define a sensible default lifecycle for the build tool encapsulated by the template.

See the elsy templates documentation for more information on using templates.

Improving elsy Performance

See the Improving Performance doc.

Contributing

See the Contributing to elsy document.

elsy's People

Contributors

joeygibson avatar lomigmegard avatar munkyboy avatar paulcichonski avatar pcavalar avatar rwhitworth avatar skoky avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

elsy's Issues

Server start command does not show server ports on "Docker for Mac"

lc server start [-prod] will normally print out something like the following when run with Docker Toolbox:

INFO[0000] starting service "prodserver"
INFO[0004] Server is running using "prodserver" service
INFO[0035] 8080/tcp is available at 172.17.8.101:32769

Where the 172.17.8.101 address comes from whatever is set for DOCKER_HOST. However DOCKER_HOST is something that is Docker Toolbox specific and Docker for Mac explicitly tells users to unset it because it is not used[1]. This breaks the showing of the IP:PORT when running Docker for Mac, the output is just:

INFO[0000] starting service "prodserver"
INFO[0004] Server is running using "prodserver" service

Need to come up with a smarter way of determining IP[2], since on Docker for Mac it is localhost of the mac.

[1] https://docs.docker.com/docker-for-mac/docker-toolbox/#/setting-up-to-run-docker-for-mac
[2] how we currently do it:

func DockerIp() (string, error) {

lc upgrade

Proposal for new command lc upgrade that updates lc to latest released version

Detect git branch that clashes with version in `lc release`

If you execute lc release --version=vx.y.z --git-commit=...., it's guaranteed to fail if you have a local branch called vx.y.z. But the failure message is not helpful. Would be nice if lc would check for this, and provide a helpful error message.

lc bootstrap no longer hiding confusing log messages for docker versions > 1.12.6

When running lc bootstrap on repos using newer versions of docker versions > 1.12.6 you will see these confusing log messages again, because docker changed their log format...again.

404 Client Error: Not Found ("repository <repo-image-name> not found: does not exist or no pull access")

I've confirmed the above on 17.06.2-ce, but I think this applies to 1.13.x too.

Use Public SBT Image in SBT Template

Our current built-in sbt template uses a non-existent image[1], which is an artifact from pre-opensourcing this project. Currently we mitigate this with docs telling users to override the default image[2], but we really need to update the sbt template to use a publicly available image.

Unfortunately there is no official sbt image so we will need to publish one.

[1] https://github.com/cisco/elsy/blob/master/template/sbt.go#L16
[2] https://github.com/cisco/elsy/blob/master/docs/templates.md#sbt

Support docker-compose v2 file format for templates

We currently only support the compose v2 file format for repos not using a built-in template (see d6b9310). This is because our templates are hardcoded into the v1 syntax and the way we are using compose file chaining does not work across files with different versions.

I propose the following:

  1. Copy the existing template files into an equivalent set of v2 templates.
  2. Update the elsy startup code to inspect the repo's compose file. If it is a v1 file, log a deprecation warning and then continue with the current code path. If it is a v2 file chain it using our v2 templates.

This gives us a clean deprecation path for the v1 templates without breaking backwards compatibility for current users.

lc teardown not removing user-defined networks

Version:

lc --version
lc version 1.2.1 (build: 1173ec7614975ce4666463731204b63b3f6095c3)

Given the following lc.yml:

project_name: sample

and docker-compose.yml:

version: '2'
services:
  blackbox-test:
    image: alpine
    command: /bin/true

Running lc ci will create the following network:

$ docker network ls
5d21e2e46f96        sample_default         bridge

It is expected that lc teardown would remove this network, but it does not. This is bad because if a user tries to later edit the ipam config for the network it will not be applied since the network is already created.

Update source files with license info

According to the APPENDIX: How to apply the Apache License to your work section of the LICENSE file, we need to put a header in each source file, with the text in that section.

sporadic memory errors during 'lc package'

This happens very infrequently (maybe 1 out of 100 times), lc package will throw the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x514c05]

goroutine 1 [running]:
panic(0x8c0fe0, 0xc8200120d0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
stash0.eng.lancope.local/dev-infrastructure/project-lifecycle/helpers.RemoveContainersOfImage(0xc82026c848, 0x7, 0x0, 0x0)
    /go/src/stash0.eng.lancope.local/dev-infrastructure/project-lifecycle/helpers/docker.go:89 +0x735
stash0.eng.lancope.local/dev-infrastructure/project-lifecycle/command.RunPackage(0xc8201fe000, 0x0, 0x0)
    /go/src/stash0.eng.lancope.local/dev-infrastructure/project-lifecycle/command/package.go:62 +0x796

Guessing this related to 82c30e4 and/or c054cde, we should probably just revert that logic since it doesn't work in all cases (i.e., large docker compose files).

Elsy Templates Incompatible with Docker-Compose File Format 2.1 and Beyond

We support the v2 docker-compose file format, but when you try to use anything beyond that (e.g., 2.1) in your repo lc bootstrap will fail with:

ERROR: Version mismatch: file /<somepath>/lc_docker_compose_template155327358 specifies version 2.0 but extension file ./docker-compose.yml uses version 2.1

This is because we hardcode version "2.0" in the template definitions, and then are not smart about dealing with minor versions. Example: https://github.com/cisco/elsy/blob/master/template/mvn.go#L79.

emberjs-u example has failing unit test

Something must have changed in one of the underlying js dependencies (which are not pinned to an exact version):

Seen with lc version v1.7.0 (build: 06ead918458493f8731f0d30a3d7830ebc665301)

lc --version
lc version v1.7.0 (build: 06ead918458493f8731f0d30a3d7830ebc665301)

cd examples/emberjs-ui/
lc teardown -f
lc ci

... 

## eventually fails on 'test' phase with:
not ok 7 PhantomJS 2.1 - Integration | Component | note list: it renders
    ---
        actual: >
            Add Note
        expected: >

        stack: >
            http://localhost:7357/assets/tests.js:226:17
            wrapper@http://localhost:7357/assets/test-support.js:7132:34
            runTest@http://localhost:7357/assets/test-support.js:3112:32
            run@http://localhost:7357/assets/test-support.js:3097:11
            http://localhost:7357/assets/test-support.js:3239:14
            process@http://localhost:7357/assets/test-support.js:2898:24
            begin@http://localhost:7357/assets/test-support.js:2880:9
            http://localhost:7357/assets/test-support.js:2940:9
        Log: |
    ...

Needs a CI build

We need a functioning CI build that will gate PRs and publish binaries for releases.

Proposal: remove features

After seeing how elsy has been used for a couple of years now, I think we should simplify it.

  • remove templates. They are hidden and feel like magic. Also hard to maintain WRT the docker-composeversion declaration
  • remove self-updating. This is brittle and unused. This task should be handled by a package manager. If anything, we should replace it with a system that occasionally reminds the user if a new version is available.
  • remove project-name option. We should promote using the standard COMPOSE_PROJECT_NAME env var.

I would like to bundle these changes into the next major version.

lc release false positive error on branch name

Seen on lc version v1.6.1 (build: bb7a94e4e4ad7e8c35dafd2f51a9e1edb6d65386)

Tried:

lc release --version=v5.1.0 --git-commit=27bbb0c6a72

Expected this to work as there was no branch or tag pre-existing with this name, but got:

There is already a branch with the name v5.1.0

It seems to have triggered that error on a branch that simply contained the v5.1.0 fragment, but did not match it exactly:

git branch -a | grep "v5.1.0"
* release-v5.1.0
 remotes/origin/release-v5.1.0

lc release should provide more useful error messages

Users have reported that if there is a branch with the same name as the new release, the error message provided by lc is not useful:

 lc release --git-commit=318642000e12833daccc8a2ddc17dbb45363f2c8 --version=v2.0.2
INFO[0000] creating, and pushing, git tag v2.0.2 at commit 318642000e12833daccc8a2ddc17dbb45363f2c8
error: src refspec v2.0.2 matches more than one.
error: failed to push some refs to 'ssh://[email protected]/ser/svc-sw-flow-sensor-agent.git'
ERRO[0000] exit status 1
ERRO[0000] command failed. use --debug to see full stacktrace

Bootstrap Fails if docker-compose.yml Only Contain Services with Repo's Image

Given an lc.yml with:

name: testbootstrap
docker_image_name: bazzer

And a docker-compose.yml with:

prodserver:
  image: bazzer

Then I would expect lc bootstrap to succeed and not pull any images. However it fails with ERROR: for prodserver pull access denied for bazzer, repository does not exist or may require 'docker login'.

Docker-compose format version support

It would be great to have support for minor version of the docker-compose template format, specifically 2.1 to get HEALTHCHECK / condition: service_healthy facilities which are very useful in general and even more when dealing witch services with warmup condition and/or post-start configs.

Add docs on what versions of docker and docker-compose elsy supports

We should publish a support matrix for both docker and docker-compose versions.

To support this we also need to test each docker/compose version pair in the matrix, the tasks involved in enabling this are:

  • Update TravisCI build to run parallel builds across the different versions
  • Figure out how to tag specific features/scenarios of the blackbox-tests if they only work starting at a given docker/compose version; so a build that runs with an earlier version can skip the feature/scenario without failing the build.

Make Templates Pluggable

We currently bake templates into the binary and only allow users to customize the docker image via lc.yml (other things can be customized through docker-compose chaining and overriding, but it is not a great user experience).

Instead we should provide users with the ability to easily introduce new templates that are curated by their organization.

Elsy should attach provenance metadata to docker image

Sometimes it is super hard to track how an image was created (i.e., git-commit, build-number). We should be attaching this provenance metadata to the image before we publish it.

One thought would be to add this using the --label arg to docker build, during the lc package phase, but this will only work for Docker 1.11 and higher.

Build Breaks with Go 1.11

With golang:1.11 the build is breaking during lc test with:

$ lc test
go test ./command ./helpers ./main ./template
# github.com/cisco/elsy/helpers
helpers/docker-compose.go:175: Debug call has possible formatting directive %q
helpers/docker.go:274: Debugf call needs 1 arg but has 2 args
ok  	github.com/cisco/elsy/command	0.013s
FAIL	github.com/cisco/elsy/helpers [build failed]
ok  	github.com/cisco/elsy/main	0.012s
ok  	github.com/cisco/elsy/template	0.006s
ERRO[0010] exit status 2
ERRO[0010] command failed. use --debug to see full stacktrace
exit status 2

golang:1.11 sha at time of writing this:

$ docker pull golang:1.11
1.11: Pulling from library/golang
Digest: sha256:4400859267b8837d7d94d4cf0c85562984ab5159d75413c23eb2f60caac0d4e4
Status: Downloaded newer image for golang:1.11

Need a Lifecycle Phase for security-scans

The use of security-scanning in the CD pipeline is getting more and more popular so we should add a lifecycle phase to lc ci called security-scan that will run the security-scan service if it is found.

`lc ci` should not report an error if no publish service defined

If you run lc ci on a dev machine, and no publish service is defined, you will see

INFO[0002] Running publish
ERRO[0003] The publish task requires that either a git branch or git tag be set, found neither
ERRO[0003] command failed. use --debug to see full stacktrace

and $? is 1. This makes it look like there's a problem, when there isn't.

I suggest we report the non-existence of a publish service or the fact that publish isn't going to run because there's no GIT_TAG env var as an INFO level message.

Custom Maven arguments when using `lc ci`

We frequently find that maven-based projects fail on Jenkins, because mvn's download optimizier decides not to check for new libraries. We've taken to manually adding lc mvn -- --update-snapshots compile (or something similar) at the front of the Jenkins job. Very annoying.

a) It would be nice if a) package/test etc commands could know the were running as lc-ci.
b) the maven template were capable of adding the --update-snapshot to the first mvn invocation.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.