deis / slugbuilder Goto Github PK

Buildpack application builder for Deis Workflow

License: MIT License

Makefile 18.09% Shell 76.18% Python 5.73%

deis-workflow heroku-buildpack kubernetes k8s unsupported

slugbuilder's Introduction

	Deis Workflow is no longer maintained. Please read the announcement for more detail.
09/07/2017	Deis Workflow v2.18 final release before entering maintenance mode
03/01/2018	End of Workflow maintenance: critical patches no longer merged
	Hephy is a fork of Workflow that is actively developed and accepts code contributions.

Deis Slugbuilder v2

Deis (pronounced DAY-iss) Workflow is an open source Platform as a Service (PaaS) that adds a developer-friendly layer to any Kubernetes cluster, making it easy to deploy and manage applications on your own servers.

For more information about the Deis Workflow, please visit the main project page at https://github.com/deis/workflow.

We welcome your input! If you have feedback, please submit an issue. If you'd like to participate in development, please read the "Development" section below and submit a pull request.

About

The slugbuilder downloads a git archive (gzipped tarball) from a specified S3 API compatible server, compiles a slug and uploads it to a specified S3 API compatible server.

This component is usually launched by the Deis Builder and used inside the Deis PaaS, but it is flexible enough to be used as a pod inside any Kubernetes cluster.

Please see https://github.com/deis/builder/blob/master/README.md for a more detailed description of how slugbuilder interacts with builder.

Development

The Deis project welcomes contributions from all developers. The high level process for development matches many other open source projects. See below for an outline.

Fork this repository
Make your changes
Submit a pull request (PR) to this repository with your changes, and unit tests whenever possible.
- If your PR fixes any issues, make sure you write Fixes #1234 in your PR description (where #1234 is the number of the issue you're closing)
The Deis core contributors will review your code. After each of them sign off on your code, they'll label your PR with LGTM1 and LGTM2 (respectively). Once that happens, the contributors will merge it

slugbuilder's People

Contributors

Stargazers

Watchers

slugbuilder's Issues

s3aws: Path not found: home/example-go:git-779ca3d7/push/slug.tgz

Tried to build deis/example-go in beta3 and got this in the logs

INFO [example-go]: rimas created initial release
INFO [example-go]: domain example-go added
INFO [example-go]: rimas deployed 779ca3d
INFO [example-go]: Cleaning up RCS for releases older than v2 (latest)
INFO [example-go]: Cleaning up orphaned environment var secrets
example-go-v2-web-uhm89 -- 2016/04/29 12:39:59 Error downloading home/example-go:git-779ca3d7/push/slug.tgz (s3aws: Path not found: home/example-go:git-779ca3d7/push/slug.tgz)
example-go-v2-web-uhm89 -- 2016/04/29 12:40:21 Error downloading home/example-go:git-779ca3d7/push/slug.tgz (s3aws: Path not found: home/example-go:git-779ca3d7/push/slug.tgz)
example-go-v2-web-uhm89 -- 2016/04/29 12:40:02 Error downloading home/example-go:git-779ca3d7/push/slug.tgz (s3aws: Path not found: home/example-go:git-779ca3d7/push/slug.tgz)

Used Minio as the object storage

USN-3464-1: Wget vulnerabilities

Package wget should be upgraded in quay.io/deis/slugbuilder image due to USN-3464-1

Proposal: containerize the development environment

This is to lower the barrier to entry for hacking on Deis while also standardizing the development environment for all contributors.

add to CI

just as other repos do, travis.ci needs to build this codebase. since the build needs to push to dockerhub and quay.io, this issue implicitly will require a makefile refactor. see https://github.com/deis/minio/pull/9/files#r46733280 for more

slugbuilder download from minio fails

here's the output from slugbuilder after a git push:

ENG000656:builder aaronschlesinger$ kd logs -f deis-slugbuilder
+ app_dir=/app
+ build_root=/tmp/build
+ cache_root=/tmp/cache
+ buildpack_root=/tmp/buildpacks
+ mkdir -p /app
+ mkdir -p /tmp/cache
+ mkdir -p /tmp/buildpacks
+ mkdir -p /tmp/build/.profile.d
+ [[ -z http://10.3.0.93:9000/git/home/rococo-seashore:git-4bad1ffa/tar ]]
+ [[ -e /var/run/secrets/object/store/access-key-id ]]
+ [[ -e /var/run/secrets/object/store/access-secret-key ]]
++ cat /var/run/secrets/object/store/access-key-id
+ keyID=8TZRY2JRWMPT6UMXR6I5
++ cat /var/run/secrets/object/store/access-secret-key
+ secretKey=gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
++ awk -F/ '{print $3}'
++ echo http://10.3.0.93:9000/git/home/rococo-seashore:git-4bad1ffa/tar
+ domain=10.3.0.93:9000
+ echo 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk 10.3.0.93:9000
+ mc --quiet config host add 10.3.0.93:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk 10.3.0.93:9000
mc: Configuration written to [/app/.mc/config.json]. Please update your access credentials.
Added host ‘10.3.0.93:9000’ successfully.
+ mc --quiet cp http://10.3.0.93:9000/git/home/rococo-seashore:git-4bad1ffa/tar /tmp/slug.tgz
mc: <ERROR> Unable to stat source ‘http://10.3.0.93:9000/git/home/rococo-seashore:git-4bad1ffa/tar’. 400 Bad Request

fix(Dockerfile): store mc binary and download on docker build

they don't have 'official' releases now, but it should be good enough to download from dl.minio.io for now

remove the directories in the env dir passed to the slugbuilder compile

To fix the issues like this deis/builder#454 which happens because of k8s creating some directories it would be better to remove these dirs before https://github.com/deis/slugbuilder/blob/master/rootfs/builder/build.sh#L171

Can't run custom slugbuilder

I'm trying to run custom slugbuilder image on my cluster, I'm following this doc and advices from slack. For now I'm trying plain unmodified slugbuilder code.

Steps to reproduce:

Env and build process:

 export DEIS_REGISTRY=quay.io/
 export IMAGE_PREFIX=szymon_szypulski
 make deploy

Image is pushed succesfully and tagged as cannary.

kubectl edit configmap slugbuilder-config -n deis. I'm changing image to quay.io/szymon_szypulski/slugbuilder:canary, pullpolicy to Always.
I'm deleting currently running builder pod, it is recreated successfully.
Pushing the app, build fails.

kubect describe pod <slugbuilder_app_pod_name> -n deis shows:

 State:              Terminated
   Reason:           Error
   Exit Code:        126

kubectl logs pod <slugbuilder_app_pod_name> -n deis` shows:
```
 /usr/bin/env: bash: Permission denied
```

I've tried minikube and vagrant based cluster.

I'm pretty sure I'm doing something wrong, but I'm not sure what :/ I assume, official images are built same way.

Add information on how to configure slugbuilder for standalone usage into the README

Notes to make, under the About section:

What to set the TAR_URL env var to
What to set the put_url env var to
How to configure the object store secret (or link to the object store docs)

This issue was created in response to #38 (comment)

cc/ @smothiki

USN-3438-1: Git vulnerability

Package git-core should be upgraded in quay.io/deis/slugbuilder image due to USN-3438-1

Proposal: Implement heroku cache using objectstore

Heroku offers a $CACHE_DIR for buildpacks to store files that will be persisted across builds (reference).

Builder offers the directory for buildpack compatibility, but the contents of the files are not persisted when the slug builder exits.

Deis V1 used the same builder for all apps and stored the cache directory in a volume, so unless it got rescheduled it cached properly.

For deis workflow however, I recommend uploading the cache dir after a build to the builder object store bucket and downloading it at the start of the next build.

Sometimes, the cache becomes corrupted and needs to be deleted. Sometimes users do not want to cache the cache directory. In that case, they could set the DEIS_BUILDPACK_CACHE environmental variable to false, which would trigger the slugbuilder to delete the cache if it does exist and skip uploading the cache at the end of the build.

download official mc client when there's an official release

This comment indicates that official releases will be ready later this month. When the release is ready, mc should be downloaded on docker build instead of checked into this repo.

Relates to deis/minio#30 and deis/builder#46.

Proposal: investigate a way to test all buildpacks

Can we add some unit or functional tests in this repo (to be run in CI) that run through all supported buildpacks? This would give us more confidence in buildpack upgrades

multi-buildpacks

Does deis workflow slugbuilder support multi-buildpacks? I want to try a special build pack at https://github.com/lastko/heroku-buildpack-edgejs Would I be able to configure primary and secondary build packs as it mentioned there?

Check the object storage endpoint to determine which signature version to use

S3 and Minio should use the S3 V4 signature version, and Google Cloud Storage in compatibility mode should use the V2 version.

Ref deis/builder#222

[Q] How to repair slug.tgz?

After recovering network issues. It was failed to deploy apps to my Deis.
The example log is the case of Node.js app.
There has package.json in the repository but Slugbuilder failed to detect it.

$ git push deis master
Counting objects: 15762, done.
Delta compression using up to 2 threads.
(snip)
Starting build... but first, coffee!
(snip)
remote:
+ unset DEIS_DEBUG
+ app_dir=/app
+ build_root=/tmp/build
+ cache_root=/tmp/cache
+ env_root=/tmp/env
+ buildpack_root=/tmp/buildpacks
+ mkdir -p /app
+ mkdir -p /tmp/cache
+ mkdir -p /tmp/env
+ mkdir -p /tmp/buildpacks
+ mkdir -p /tmp/build/.profile.d
+ [[ -z home/arsdk4-5-2:git-808801e5/tar ]]
+ get_object
+ tar -xzf /tmp/slug.tgz -C /app/
+ unset TAR_PATH
+ [[ '' == \- ]]
+ slug_file=/tmp/slug.tgz
+ '[' -d /tmp/app ']'
+ cp -r /app/. /tmp/build
+ export APP_DIR=/app
+ APP_DIR=/app
+ export HOME=/app
+ HOME=/app
++ openssl rand -base64 32
+ REQUEST_ID=pbq6+PVI0B21ZkAYWpOK97lRPSJbA8XmEdlCQD8ZC6g=
+ export REQUEST_ID
+ export STACK=cedar-14
+ STACK=cedar-14
+ [[ -n '' ]]
+ buildpacks=($buildpack_root/*)
+ selected_buildpack=
+ [[ -n '' ]]
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-clojure.git/bin/detect /tmp/build
+ buildpack_name=no
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-go.git/bin/detect /tmp/build
+ buildpack_name=
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-gradle.git/bin/detect /tmp/build
+ buildpack_name=no
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-grails.git/bin/detect /tmp/build
+ buildpack_name=no
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-java.git/bin/detect /tmp/build
+ buildpack_name=no
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-multi.git/bin/detect /tmp/build
+ buildpack_name='no .buildpacks file'
+ for buildpack in '"${buildpacks[@]}"'
++ /tmp/buildpacks/heroku-buildpack-nodejs.git/bin/detect /tmp/build
+ buildpack_name=
(snip)
+ echo_title 'Unable to select a buildpack'
+ output_redirect
----->' 'Unable to select a buildpack'
+ [[ /tmp/slug.tgz == \- ]]
+ cat -
+ exit 1
+ sleep_before_exit
+ sleep 3
-----> Unable to select a buildpack
(snip)

Upgrade to the new object-storage-cli version

Since deis/object-storage-cli#12 is done, this component should use the new version to suppress debug output when DEIS_DEBUG is set

Deploy hung waiting for slugbuilder pod exit

Placeholder issue, will see if this comes back.

Deploy hung, slugbuilder pod also appeared hung. had to manually delete the pod for the deploy to continue.

[master 8e9380f] Bump
Counting objects: 520, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (515/515), done.
Writing objects: 100% (520/520), 53.75 KiB | 0 bytes/s, done.
Total 520 (delta 473), reused 0 (delta 0)
Starting build... but first, coffee!
-----> Go app detected



-----> Checking Godeps/Godeps.json file.
-----> Installing go1.4.2... done
-----> Running: godep go install -tags heroku ./...
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 1.7M
Build complete.
Launching app.
Launching...
Done, earthy-instinct:v209 deployed to Deis

Use 'deis open' to view this application in your browser

To learn more, use 'deis help' or visit http://deis.io

To ssh://[email protected]:2222/earthy-instinct.git
 * [new branch]      master -> master

real    39m19.671s
user    0m0.050s
sys 0m0.054s

k:beef ~ $ kd get po
NAME                                          READY     STATUS    RESTARTS   AGE
deis-builder-fp4d8                            1/1       Running   0          2d
deis-database-z0j7q                           1/1       Running   0          3d
deis-logger-fluentd-ug070                     1/1       Running   0          3d
deis-logger-fluentd-yl5wi                     1/1       Running   0          3d
deis-logger-l6ajn                             1/1       Running   0          3d
deis-minio-r15ow                              1/1       Running   0          3d
deis-registry-rccnn                           1/1       Running   0          3d
deis-router-0t5gf                             1/1       Running   0          3d
deis-workflow-c2a3i                           1/1       Running   0          20h
slugbuild-earthy-instinct-8e9380f2-87440faf   1/1       Running   0          37m
k:beef ~ $ kd logs slugbuild-earthy-instinct-8e9380f2-87440faf
-----> Go app detected
-----> Checking Godeps/Godeps.json file.
k:beef ~ $ kd delete po slugbuild-earthy-instinct-8e9380f2-87440faf
pod "slugbuild-earthy-instinct-8e9380f2-87440faf" deleted

Use deis-objectstorage-secret.yaml to get S3 credentials

Object storage docs should also be upgraded when this work is done.

Related: deis/workflow#61

Could not create the Namespace in Kubernetes - config:push

I'm using deis config:push to send the environment variables from a .env file to my application and it's giving me

Creating config... ...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��...��o..��.o.��..o��Error: Unknown Error (400): {"detail":"Could not create the Namespace in Kubernetes"}

How is that that I cannot create a namespace?

Very long deployment

Hi,

Environment :
Deis: current deis-dev
CoreOS : 976.0.0
Platform : AWS with kube-aws

When deploying an app using a buildpack it takes a very long time before actually starting to run.

 ~/D/deis-aws  kubectl get pods --namespace=deis                                                                         
NAME                                 READY     STATUS    RESTARTS   AGE
deis-builder-1iuw8                   1/1       Running   4          16m
deis-controller-diqmy                1/1       Running   5          16m
deis-database-mv8o5                  1/1       Running   0          16m
deis-minio-7dk0j                     1/1       Running   0          16m
deis-registry-xa38b                  1/1       Running   15         16m
deis-router-y0btr                    1/1       Running   0          16m
deis-workflow-manager-8xmk5          1/1       Running   0          16m
slugbuild-george-dfb1a772-c5df21bb   0/1       Pending   0          8m

And here's where the client-side git push get stuck :

 ~/D/deis-aws  git push deis master
Counting objects: 68, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (63/63), done.
Writing objects: 100% (68/68), 40.79 KiB | 0 bytes/s, done.
Total 68 (delta 31), reused 0 (delta 0)
remote: 2016/03/13 12:31:18 Running in debug mode
Running git hook
read [0000000000000000000000000000000000000000,dfb1a772d5a294048808fcd6d04f3485e0728f50,refs/heads/master]
Controller request /v2/hooks/push (body elided)
Controller request POST /v2/hooks/config
{"receive_user":"admin","receive_repo":"george"}
got the following config back for app george: {Owner:admin App:george Values:map[] Memory:map[] CPU:map[] Tags:map[] UUID:053bf621-5d9d-4675-bcf6-21a25f35f9b8 Created:2016-03-13 12:30:56 +0000 UTC Updated:2016-03-13 12:30:56 +0000 UTC}
running [git archive --format=tar.gz --output=george.tar.gz dfb1a772] in directory /home/git/george.git
running [tar -xzf george.tar.gz -C /home/git/george.git/build/tmp825703070/] in directory /home/git/george.git
Uploading tar to http://10.3.0.6:9000/git/home/george:git-dfb1a772/tar
Starting build... but first, coffee!
Starting pod slugbuild-george-dfb1a772-c5df21bb
Pod spec: {
  "metadata": {
    "name": "slugbuild-george-dfb1a772-c5df21bb",
    "namespace": "deis",
    "creationTimestamp": null,
    "labels": {
      "heritage": "slugbuild-george-dfb1a772-c5df21bb"
    }
  },
  "spec": {
    "volumes": [
      {
        "name": "minio-user",
        "secret": {
          "secretName": "minio-user"
        }
      }
    ],
    "containers": [
      {
        "name": "deis-slugbuilder",
        "image": "quay.io/deisci/slugbuilder:v2-beta",
        "env": [
          {
            "name": "DEBUG",
            "value": "1"
          },
          {
            "name": "TAR_URL",
            "value": "http://10.3.0.6:9000/git/home/george:git-dfb1a772/tar"
          },
          {
            "name": "put_url",
            "value": "http://10.3.0.6:9000/git/home/george:git-dfb1a772/push"
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "minio-user",
            "readOnly": true,
            "mountPath": "/var/run/secrets/object/store"
          }
        ],
        "imagePullPolicy": "Always"
      }
    ],
    "restartPolicy": "Never",
    "serviceAccountName": ""
  },
  "status": {}
}

After 10 minutes approximately the "slugbuild-george-dfb1a772-c5df21bb" pod disappeared and the build continued.

Why does it take so long ? :/

Publish immutable artifact tags + canary version to Docker registries

We want to have the ability to have deis/charts reference a specific immutable image which will enable us better traceability on issues that come into various components.

What we've done with efforts in other repositories is publish a version:

git-[short sha] that will never be overwritten
canary that will move to reference the latest version available of a component

Examples of the easiest way to roll this change out can be seen in deis/workflow#487, deis/builder#246, and deis/workflow-manager#11

Run docker build on all PR builds

Similar to what builder does in its .travis.yml file, slugbuilder should run make ... docker-build to ensure that the Dockerfile is correct

Failed git push output isn't always displayed correctly

Similarly to deis/dockerbuilder#76, when using a broken buildpack sometimes the output is orderly and useful, and sometimes we get a raw k8s dump from a pod exiting through builder.

Expose DEBUG env variable for for Kubernetes manifests

As a user of slugbuilder, I'd like to be able to configure the DEBUG level (currently: true vs. false) via a Kubernetes manifest pod ENV variable. See how Builder allows this behavior:

https://github.com/deis/charts/blob/master/deis-dev/manifests/deis-builder-rc.yaml#L35

pre-compile/post-compile hooks do not have PATH set as expected

I tried to create the following post-commit hook with a Node.js buildpack app:

#!/bin/sh
echo "$PWD"
npm run deis-post-compile

Which fails with the following error:

/
/tmp/build/bin/post-compile: 3: /tmp/build/bin/post-compile: npm: not found

I was expecting PWD to be set to /tmp/build/ or /tmp/build/bin/
I was expecting to be able to call node or npm

remove extraneous `mc` output

Follows deis/builder#60

Slugbuilder broken on empty default process types

Heroku's java buildpack's bin/release script used to return:


---
addons:
  - heroku-postgresql

Now it returns:

---

causing the error

-e:1:in `<main>': undefined method `[]' for nil:NilClass (NoMethodError)

in the ruby script here.

Support for non-git buildpacks

Currently BUILDPACK_URL has to be a git repo.

There is (at least one) a buildpack which is distributed in pre-complied form (*.tgz). It's possible to pass link to the archive in heroku, buildpack in git requires some manual action (complication), so it won't work in Workflow.

Is it possible to re-implement this in Workflow?

As workaround I've unpacked provided archive and pushed it to separate git repo.

Error when compiling a GO app

This error is intermittently occurring when trying to build a go app (example go) .
remote: �+ echo '�----->' Checking Godeps/Godeps.json file. remote: �-----> Checking Godeps/Godeps.json file. remote: �+ read line remote: � remote: �gzip: stdin: unexpected end of file remote: �tar: Unexpected EOF in archive remote: �tar: Unexpected EOF in archive remote: tar: Error is not recoverable: exiting now
This is happening because the go slug is not properly downloading the go (https://github.com/heroku/heroku-buildpack-go/blob/master/bin/compile#L160)

Improve testing for Slugbuilder

As of now slugbuilder is a part of full E2E test. But if we look at functionality here slugbuilder only does. two things in DEIS. Takes a tar file extracts the code compiles and generates slug file.
Any PR in slugbuilder should or also do this test instead of running entire E2E suite.

refer deis/workflow-e2e#303

Download object-storage-cli from GCS

After deis/object-storage-cli#15 is complete, object-storage-cli builds will be available in GCS. This issue is for downloading those builds from GCS instead of Bintray

Depends on deis/object-storage-cli#15