buildkite / cli Goto Github PK
View Code? Open in Web Editor NEWA command line interface for Buildkite.
License: MIT License
A command line interface for Buildkite.
License: MIT License
I love that the app is very explicit about tokens and securely storing them, and tells me that it "Securely stored GraphQL token! πͺ"
But I'm puzzled about where they are stored, and how to remove them later. Maybe "Securely stored GraphQL token in your Keychain! πͺ" or something?
I am coming from bksr for local run where a default bksr would allow you to choose step from your pipeline.yml.
Can the default for local run retain the same behavior? Currently bk local run
doesnt (appear to) allow you to choose or specify the specific step.
Consider this pipeline:
https://gist.github.com/drnic/a042bd719ed9213c5af7fd1605f8bf93#file-pipeline-bk-run-docker-fail-yml
When you run it, the -E AGENT_ENV=foobar
value is not propagated into Docker:
git clone https://gist.github.com/drnic/a042bd719ed9213c5af7fd1605f8bf93 ~/workspace/pipeline.bk-run-docker-fail
cd ~/workspace/pipeline.bk-run-docker-fail
bk run pipeline.bk-run-docker-fail.yml -E AGENT_ENV=where-is-this-env
The output should include the two lines:
STEP_ENV=found-step-env
AGENT_ENV=where-is-this-env
But it only includes the first one:
STEP_ENV=found-step-env
Why does bk local run -E AGENT_ENV=xyz
not pass $AGENT_ENV
into the docker plugin env?
Version: d57d7d87d471c114eaf0f6ab91743568abad56c7
I have this in my pipeline.yml file:
steps:
- label: foo
command: 'echo hi'
plugins:
- docker-compose#${COMPOSE_VERSION}:
run: app
And I run this:
$ export COMPOSE_VERSION=v2.6.0
$ bk local run
2019/02/06 10:19:41 Error getting git commit: exit status 128
2019/02/06 10:19:41 Error getting git branch: exit status 128
>>> Starting local agent π€
>>> Starting build π
>>> Executing initial command: buildkite-agent pipeline upload
>>> Executing command step foo
~~~ Setting up plugins
$ cd /Users/ssuter/workspace/bktest
π¨ Error: Failed to checkout plugin docker-compose: Incomplete plugin path "docker-compose"
>>> π¨ Command failed in 3.069498517s
π¨ Job failed with code 1
Hi there,
Thank you very much for implementing buildkite cli.
Currently, I got an issue running bk browse
.
The pipeline can be found by running
bk pipline list --fuzzy=pipeline-name
However, when im in pipeline-name
directory and run
bk browse
It outputs an error
π¨ Pipeline doesn't exist
This is the debug version of the command
2019/09/16 10:18:16 [keyring] Considering backends: [pass file]
2019/09/16 10:18:16 [keyring] Failed backend pass: The pass program is not available
2019/09/16 10:18:16 [keyring] Expanded file dir to /home/bangn/.buildkite/keyring/
2019/09/16 10:18:16 [keyring] Expanded file dir to /home/bangn/.buildkite/keyring/
Enter passphrase to unlock /home/user/.buildkite/keyring/:
π¨ Pipeline doesn't exist
Im using bk
cli version 1.0.0
on debian 10.
Do you have any idea on why it cannot browse
to the pipeline?
Cheers,
Bang
https://buildkite.com/docs/pipelines/conditionals
steps:
- command: "echo --- All good. Nice one."
label: ":git:"
if: build.branch == 'master'
- command: "echo --- ERROR this shouldn't run; exit 1"
label: ":git:"
if: build.branch != 'master'
When I run this pipeline.yml it runs both steps; but it should only run one or the other.
@keithpitt is this something that should work, or one of those "its annoying to implement everything in ruby + golang" (my words not yours) gaps ;)
GoBinaries is an awesome service that lets non-go users start using go-developed tools right away.
β curl -sf https://gobinaries.com/buildkite/cli/cmd/bk | sh
==> Downloading github.com/buildkite/cli/cmd/bk@master
==> Resolved version master to v1.0.0
==> Downloading binary for linux amd64
==> Installing bk to /usr/local/bin
==> Installation complete
I'm running Linux Subsystem on Windows, and when running bk configure
, I get the below error, since WLS isn't able to open a browser. Is it possible to output the URL so that I can copy and paste into the browser. (Similar behaviour to how composer handles this.)
We need to authorize this app to access your repositories. This authorization is stored securely locally, buildkite.com never gets access to it.
When you press enter, your default browser will open and authenticate to github.com
β
π¨ Github OAuth error: exec: "xdg-open": executable file not found in $PATH```
Running 0.3.0
Somewhat related to #11
In my case, I've got a Rails app (with Docker) and develop locally (also with Docker). Sometimes I build something and CI fails way more than excepted. π°
What I'd love to be able to do is to locally grab just the output of the failures for one step of our builds. From there I could pass this to RSpec's --only-failures
option and easily iterate on anything broken.
On linux, attempting to specify a secret-backend other than kwallet (which isn't installed, because people tend to not use KDE here) fails:
# me on my-box in ~/mydir on git:master βοΈ [10:04:40]
β go run cmd/bk/main.go configure buildkite
ββββ Configuring Buildkite GraphQL credentials
Create a GraphQL token at https://buildkite.com/user/api-access-tokens/new. Make sure to tick the GraphQL scope at the bottom.
GraphQL Token: [email protected] β
οΏ½οΏ½ The name org.kde.kwalletd was not provided by any .service files
exit status 1
[1] 2541 exit 1 go run cmd/bk/main.go configure buildkite
# me on my-box in ~/mydir on git:master βοΈ [10:06:01]
β go run cmd/bk/main.go configure buildkite --keyring-backend=secret-service
ββββ Configuring Buildkite GraphQL credentials
Create a GraphQL token at https://buildkite.com/user/api-access-tokens/new. Make sure to tick the GraphQL scope at the bottom.
GraphQL Token: [email protected] β
οΏ½οΏ½ The name org.kde.kwalletd was not provided by any .service files
exit status 1
[1] 2918 exit 1 go run cmd/bk/main.go configure buildkite --keyring-backend=secret-service
# me on my-box in ~/mydir on git:master βοΈ [10:06:44]
β go run cmd/bk/main.go configure buildkite --keyring-backend=file
ββββ Configuring Buildkite GraphQL credentials
Create a GraphQL token at https://buildkite.com/user/api-access-tokens/new. Make sure to tick the GraphQL scope at the bottom.
GraphQL Token: [email protected] β
οΏ½οΏ½ The name org.kde.kwalletd was not provided by any .service files
exit status 1
[1] 3048 exit 1 go run cmd/bk/main.go configure buildkite --keyring-backend=file
# me on my-box in ~/mydir on git:master βοΈ [10:07:04]
β go run cmd/bk/main.go --keyring-backend=file configure backend
οΏ½οΏ½ unexpected backend
exit status 1
[1] 3225 exit 1 go run cmd/bk/main.go --keyring-backend=file configure backend
# me on my-box in ~/mydir on git:master βοΈ [10:07:09]
β go run cmd/bk/main.go --keyring-backend=file configure buildkite
ββββ Configuring Buildkite GraphQL credentials
Create a GraphQL token at https://buildkite.com/user/api-access-tokens/new. Make sure to tick the GraphQL scope at the bottom.
GraphQL Token: [email protected] β
οΏ½οΏ½ The name org.kde.kwalletd was not provided by any .service files
exit status 1
[1] 3378 exit 1 go run cmd/bk/main.go --keyring-backend=file configure buildkite
It would be very useful to be able to retry a failed job locally via bk
by referencing the BUILDKITE_JOB_ID
. I'd imagine bk
fetching the job details from the API and starting a local build with the same environment variables, etc.
$ brew upgrade
...
==> Upgrading buildkite/cli/bk v1.0.0 -> v1.1.0
==> Downloading $STABLE_URL
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.
##O=# #
curl: (6) Could not resolve host: $STABLE_URL
Error: Failed to download resource "bk"
Download failed: $STABLE_URL
Looks like https://github.com/buildkite/cli/blob/master/.github/tap-release.yml#L11 $STABLE_URL
and $STABLE_SHA256
are not being replaced with a value; but $STABLE_VERSION
is correctly being given a value.
My local bk
formula is:
# Generated by https://github.com/buildkite/cli/blob/master/.github/tap-release.yml
class Bk < Formula
homepage "https://github.com/buildkite/cli"
desc "A command line interface for Buildkite."
url "$STABLE_URL"
version "v1.1.0"
sha256 "$STABLE_SHA256"
def install
mv "bk-darwin-amd64-1.1.0", "bk"
bin.install "bk"
end
test do
assert_match "bk #{version}", shell_output("#{bin}/bk --version 2>&1")
end
end
CLI version: 0.3.0
OS: Arch linux
β bk init --pipeline=research
Checking for pipeline file: .buildkite/pipeline.yml β
Checking for git repository and remote: https://github.com/quantopian/research_env β
Enter passphrase to unlock /home/irl/.buildkite/keyring/:
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/buildkite/cli.InitCommand(0xa67ac0, 0xe56ae8, 0xa65020, 0xc00017dd70, 0x0, 0x0, 0x0, 0x7ffe0394a4e7, 0x8, 0x7f3124a12290, ...)
/Users/lachlan/go/src/github.com/buildkite/cli/cmd_init.go:99 +0x1c5f
main.run.func5(0xc0001ea000, 0x409dc3, 0x936360)
/Users/lachlan/go/src/github.com/buildkite/cli/cmd/bk/main.go:192 +0xb5
gopkg.in/alecthomas/kingpin%2ev2.(*actionMixin).applyActions(0xc0001b0498, 0xc0001ea000, 0x0, 0x0)
/Users/lachlan/go/pkg/mod/gopkg.in/alecthomas/[email protected]/actions.go:28 +0x6d
gopkg.in/alecthomas/kingpin%2ev2.(*Application).applyActions(0xc0001e4690, 0xc0001ea000, 0x0, 0x0)
/Users/lachlan/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:557 +0xdf
gopkg.in/alecthomas/kingpin%2ev2.(*Application).execute(0xc0001e4690, 0xc0001ea000, 0xc000176b50, 0x1, 0x1, 0x0, 0x0, 0x1, 0xc00019c2e8)
/Users/lachlan/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:390 +0x8f
gopkg.in/alecthomas/kingpin%2ev2.(*Application).Parse(0xc0001e4690, 0xc0000a0010, 0x3, 0x3, 0x1, 0xc00019c2b8, 0x0, 0x1)
/Users/lachlan/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:222 +0x1fa
main.run(0xc0000a0010, 0x3, 0x3, 0x9f9d88)
/Users/lachlan/go/src/github.com/buildkite/cli/cmd/bk/main.go:356 +0x2a16
main.main()
/Users/lachlan/go/src/github.com/buildkite/cli/cmd/bk/main.go:21 +0x6e
Without specifying pipeline:
β bk init
Checking for pipeline file: .buildkite/pipeline.yml β
Checking for git repository and remote: https://github.com/quantopian/research_env β
Enter passphrase to unlock /home/irl/.buildkite/keyring/:
Checking for buildkite pipeline: Created https://buildkite.com/quantopian/research-env β
Checking github repository config for quantopian/research_env: Created webhook β
Ok! Your project is ready to go at https://buildkite.com/quantopian/research-env π
It would be ideal to be able to provide either a config file (yml, toml, or other) to the bk init
command to configure the pipeline from the CLI, or even better a bk pipeline config
or similar command to do so.
That way we can create a "standard" pipeline config and easily bootstrap a bunch of repos. eg: GH settings, skip settings, etc.
Hi there, I'm attempting to run a pipeline locally on Windows, and am having some trouble. I've pasted the output below and you can find a copy of the static steps here.
Any ideas where I'm going wrong?
Thanks
Ξ» bk local run .buildkite/premerge.steps.yaml --debug
2019/02/06 18:29:18 [keyring] Considering backends: [file]
2019/02/06 18:29:18 Serving API on http://127.0.0.1:49256
2019/02/06 18:29:18 Creating bootrap script at C:\Users\JAMIEB~1\AppData\Local\Temp\bootstrap-450177923
>>> Starting local agent π€
>>> Starting build π
>>> Executing initial command: buildkite-agent pipeline upload ".buildkite/premerge.steps.yaml"
_ _ _ _ _ _ _ _
| | (_) | | | | (_) | | |
| |__ _ _ _| | __| | | ___| |_ ___ __ _ __ _ ___ _ __ | |_
| '_ \| | | | | |/ _` | |/ / | __/ _ \ / _` |/ _` |/ _ \ '_ \| __|
| |_) | |_| | | | (_| | <| | || __/ | (_| | (_| | __/ | | | |_
|_.__/ \__,_|_|_|\__,_|_|\_\_|\__\___| \__,_|\__, |\___|_| |_|\__|
__/ |
http://buildkite.com/agent |___/
2019-02-06 18:29:18 NOTICE Starting buildkite-agent v3.8.4 with PID: 40328
2019-02-06 18:29:18 NOTICE The agent source code can be found here: https://github.com/buildkite/agent
2019-02-06 18:29:18 NOTICE For questions and support, email us at: [email protected]
2019-02-06 18:29:18 INFO Configuration loaded from: C:\buildkite-agent\buildkite-agent.cfg
2019-02-06 18:29:18 DEBUG Bootstrap command: C:\Users\JAMIEB~1\AppData\Local\Temp\bootstrap-450177923
2019-02-06 18:29:18 DEBUG Build path: C:\Users\JAMIEB~1\AppData\Local\Temp\buildkite-build-735988486
2019-02-06 18:29:18 DEBUG Hooks directory: C:\buildkite-agent\hooks
2019-02-06 18:29:18 DEBUG Plugins directory: C:\buildkite-agent\plugins
2019-02-06 18:29:18 INFO Running builds within a pseudoterminal (PTY) has been disabled
2019-02-06 18:29:18 INFO Registering agent with Buildkite...
2019-02-06 18:29:18 INFO Started 1 Agent(s)
2019-02-06 18:29:18 INFO You can press Ctrl-C to stop the agents
2019-02-06 18:29:18 DEBUG POST http://127.0.0.1:49256/register
2019-02-06 18:29:18 WARN Post http://127.0.0.1:49256/register: dial tcp 127.0.0.1:49256: socket: The requested service provider could not be loaded or initialized. (Attempt 1/30 Retrying in 10s)
2019-02-06 18:29:28 DEBUG POST http://127.0.0.1:49256/register
2019-02-06 18:29:28 WARN Post http://127.0.0.1:49256/register: dial tcp 127.0.0.1:49256: socket: The requested service provider could not be loaded or initialized. (Attempt 2/30 Retrying in 10s)
2019/02/06 18:29:30 [http] GET /register 127.0.0.1:49257
>>> π An unknown agent API endpoint was requested (GET /register).
File an issue at https://github.com/buildkite/cli/issues and we'll see what we can do!
2019/02/06 18:29:30 [http] GET /favicon.ico 127.0.0.1:49257
>>> π An unknown agent API endpoint was requested (GET /favicon.ico).
File an issue at https://github.com/buildkite/cli/issues and we'll see what we can do!
2019-02-06 18:29:38 DEBUG POST http://127.0.0.1:49256/register
2019-02-06 18:29:38 WARN Post http://127.0.0.1:49256/register: dial tcp 127.0.0.1:49256: socket: The requested service provider could not be loaded or initialized. (Attempt 3/30 Retrying in 10s)
2019-02-06 18:29:48 DEBUG POST http://127.0.0.1:49256/register
2019-02-06 18:29:48 WARN Post http://127.0.0.1:49256/register: dial tcp 127.0.0.1:49256: socket: The requested service provider could not be loaded or initialized. (Attempt 4/30 Retrying in 10s)
2019-02-06 18:29:58 DEBUG POST http://127.0.0.1:49256/register
2019-02-06 18:29:58 WARN Post http://127.0.0.1:49256/register: dial tcp 127.0.0.1:49256: socket: The requested service provider could not be loaded or initialized. (Attempt 5/30 Retrying in 10s)
2019-02-06 18:30:08 DEBUG POST http://127.0.0.1:49256/register
2019-02-06 18:30:08 WARN Post http://127.0.0.1:49256/register: dial tcp 127.0.0.1:49256: socket: The requested service provider could not be loaded or initialized. (Attempt 6/30 Retrying in 10s)
>>> Gracefully shutting down...
π¨ Initial command failed: context canceled
I notice the 'Create a build' REST API call includes the capability to set environment variables.
It would be awesome if this CLI app exposed this feature so we could pass environments variables to builds from the command line.
I have a step that fetches meta-data, but bk run local
fails.
My bash script invokes:
CF_ORGANIZATION=$(buildkite-agent meta-data get cf-organization)
CF_SPACE=$( buildkite-agent meta-data get cf-space)
But bk
fails with:
ci/scripts/cf-push.sh: line 5: buildkite-agent: command not found
I do have buildkite-agent
installed locally. How do I get bk
to use it?
I create a GraphQL token for my org but I don't give it any scopes (because I don't know who/when they are needed, and there are so many of them, so I just ignore it).
Then when I run bk init
I get a GraphQL error: Authentication required
error:
$ bk configure
...
$ bk init
Checking for pipeline file: .buildkite/pipeline.yml β
Checking for git repository and remote: https://github.com/drnic/buildkite-envvar-playtime β
Checking for buildkite pipeline: GraphQL error: Authentication required. Please supply a valid API Access Token: https://buildkite.com/docs/apis/graphql-api#authentication β
π¨ GraphQL error: Authentication required. Please supply a valid API Access Token: https://buildkite.com/docs/apis/graphql-api#authentication
My guess is that it isn't an authentication issue (authentication = who is this?) but an authorization issue (is this user allowed to do operation xyz?)
Is it possible for the bk init
command error nicely and educate me in future?
Which scopes do I need for bk init
and other bk
operations?
Is all we need a list of repos? There's a new GitHub App permissions scope that can give you that, without all this:
If we are asking for a god token, perhaps we need to spell out why a little more?
Also perhaps we need to say that the GitHub authorisation is only valid to your cli, and can't be used outside of the cli (if we were, say, to update the GitHub App settings or something). Assuming that's correct? I was only a little more comfortable once I noticed the 127.0.0.1
redirect at the bottom of the authorise screen.
On macOS I'm trying to use the keychain
backend, but file
is always the one chosen even when I explicitly specify --keyring-backend=keychain
.
This seems to have been introduced in 3e9370c. Following that change, inferring the file
backend from keyringFileDir
is no longer valid because the latter string now defaults to non-empty.
As confirmation of this behaviour, I'm able to work around the problem and use keychain
by specifying --keyring-file-dir ""
.
I think the solution is to only try to infer a backend if it's not explicitly specified. The inference of file
from keyringFileDir
will also need rethinking.
Hello BK :)
Using the bk init function to set up scaffolding and pushing up the skeleton init commit results in the agent running infinite builds.
I used the client to manually init a repo I had to test the bk agent using the AWS stack set up. When I checked the bk pipeline I noticed that it was continually building and had to manually stop the build.
I am happy to put in a fix for this - any preferences?
currently bk
seems to point only at github.com... any chance there will be support for GitHub Enterprise and on-prem installations?
Thanks!
I'm unable to successfully create a build on osx High Sierra. I've tried using both the keyring and a file backend but I don't have enough documentation and the error messages are too opaque to know what I'm doing wrong. Please help:
> bk configure
ββββ Ok! Let's get started with configuring bk π
ββββ Configuring Buildkite GraphQL credentials
Create a GraphQL token at https://buildkite.com/user/api-access-tokens/new. Make sure to tick the GraphQL scope at the bottom.
GraphQL Token: [redacted] β
Securely stored GraphQL token! πͺ
Wrote configuration to /Users/[redacted]/.buildkite/config.json π
For now, we will assume you are using GitHub. Support for others is coming soon! π
ββββ Let's configure your github.com credentials π»
We need to authorize this app to access your repositories. This authorization is stored securely locally, buildkite.com never gets access to it.
When you press enter, your default browser will open and authenticate to github.com
Authenticated as [redacted] β
Securely stored GitHub token! πͺ
Ok, you are good to go!
^^ This seems to work and I've also tried the following commands using both the osx keychain and a file keychain so I don't think this is the issue.
> bk init
Checking for pipeline file: .buildkite/pipeline.yml β
Checking for git repository and remote: exit status 128 β
π¨ exit status 128
> git status
On branch uavcan-v1/issues/235
nothing to commit, working tree clean
As you can see, this is a fully functional repository that I use to submit to github all the time.
> bk build create --debug --debug-graphql
2019/07/27 11:05:04 [keyring] Considering backends: [keychain pass file]
2019/07/27 11:05:04 [keyring] Querying keychain for service="buildkite", account="graphql-token", keychain=""
2019/07/27 11:05:04 [keyring] Found item "Buildkite GraphQL Token"
π¨ exit status 128
Again, no useful debug information here, just "exit status 128".
Other commands:
> bk browse
π¨ exit status 128
> bk pipeline list
uavcan/libuavcan-v1
> bk local run
(this works)
I'd really like the ability to initiate a build from local changes! Thanks for helping (You all are rockstarsπΈ by the way).
Hi,
I'm sorry if I'm not posting in the right place but I don't seem to be able to figure out how to do this:
I am using Buildkite on OsX to build ReactNative apps. All good and I'm very happy but I have the case where I need to pass some environment variables to the agent that are not the same with the machine I'm working on.
For example ANDROID_HOME
Setting this in the pipeline file is not an option as I want to be able to build the code without the need of extra changes and I don't know how to pass to the agent the variables already existent to a "normal" user.
Thanks for any help you could give π€
Hi there, I've got a steps file which includes steps with elements in the env
field:
- label: "build :android:"
command: bash -c ci/build.sh
<<: *common
artifact_paths:
- logs/**/*
env:
WORKER_TYPE: "AndroidClient"
BUILD_TARGET: "local"
SCRIPTING_TYPE: "mono"
However, when I run bk-cli local run steps.yaml
this step fails with:
ci/build.sh: line 8: WORKER_TYPE: unbound variable
Is this the intended behaviour?
OS: macOS 10.13.6
CLI version: 0.4.0
buildkite-agent version: 3.8.4, build 2783
Hey!
We have a handful of folks within our org who would like to use the local pipeline running functionality of this, however they need to be running an instance of the agent, which requires our agent token. I'm hesitant to give out our org's agent token to every developer who wants to do local testing, given that it allows registration of new agents, which may unintentionally cause issues. Is there any way to get the local pipeline functionality without giving out our agent token?
I tried using a fake token to see if it would still work, but unfortunately it appears to error out if the token is invalid.
What is bk
's relationship with https://github.com/buildkite/emojis?
In the example pipeline below, only :thumbsup:
is supported by bk
:
steps:
- command: "sleep 2"
label: ":thumbsup:"
- command: "sleep 2"
label: ":thumbsup::skin-tone-2:"
- command: "sleep 2"
label: ":+1:"
- command: "sleep 2"
label: ":+1::skin-tone-2:"
The output includes:
>>> Executing command step π
...
>>> Executing command step π:skin-tone-2:
...
>>> Executing command step :+1:
...
>>> Executing command step :+1::skin-tone-2:
I should be able to override the Buildkite org for init
- it assumes it will match the GitHub org.
Because of crippling performance issues with Docker on macOS Mojave, I use dinghy to run Docker on macOS, which essentially runs it in a VM then mounts local directories over NFS into the VM. It's hacky, but it is about a 10x performance improvement in IO-heavy projects :(.
I'm not sure if there are inherent limitations that would make this impossible, but it'd be great if I could use bk
in this context. Currently, I get the following bk local run
error:
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
After successfully run bk configure
, any other commands such bk browse
errors with message:
π¨ %!!(MISSING)v(PANIC=runtime error: invalid memory address or nil pointer dereference)
Running on MacOs 10.14.3
Installed using homebrew
bk --version
0.3.0
Currently, when running bk local run
with a step containing skip: true
the agent still runs the commands
.buildkite/pipeline.yml
:steps:
- label: ":coffee: i need coffee :coffee:"
skip: true
commands:
- echo "coffee is better than tea" > file.out
- wait
- label: ":hammer: file.out is created"
commands:
- echo "There shouldn't be a file.out but there is :("
bk local run
and observe the creation of the file.out
which shouldn't exist as it should be skippedThe key skip
is respected, with its bool
value or string
I often want to download all artifacts for a particular build or job to my filesystem for local build pipeline debugging. It'd be neat if bk download artifacts https://buildkite.com/acme/widgets/builds/123
worked, or something.
We have a monorepo with a pre-merge pipeline that runs many disparate steps.
For local iteration it would be really convenient to run a subset of those steps.
Could bk local run ..... --steps foo bar baz
become possible? Run the specified step(s) in the order given?
Whilst discovering buildkite/agent#1181 I came across the desire for bk run
or bk local run
to split out its behaviour and run a local pipeline, but use remote agents.
I like bk local run
-- it allows me to test a new pipeline before I "commit" all future PRs/commits/team members to the pipeline changes. You can iterate on the pipeline.
The command happens to run the steps locally; though to me this is tangential.
I can imagine this is not a small PR. I wanted to raise the topic and perhaps someone can contribute if its a good or bad idea.
In trying to initialize a new pipeline on Buildkite, I was attempting to run the bk init
command.
Nothing had been setup in BuildKite at this stage.
When I ran the command at a user level (just part of the default "Everyone" team), the command failed with:
"Checking for buildkite pipeline: GraphQL error: Please ensure at least one team is added to this pipeline"
When running as an admin-level user, the init succeeded and I noticed when going to the web ui no team was assigned.
So it seems two things are happening:
Also, just to note: a user just part of the default everyone group, is able to select a team and successfully make a pipeline in the web ui, so I don't believe it's some sort of permissions issue. Though maybe there is some permission we can provide users so they can...not pass a team?
As applied on: 99designs/aws-vault@cb88446 we should update the keyring
deps to include this fix: 99designs/keyring#52 otherwise kwallet
backend does not work.
What is the licensing on this repository? Is it MIT like the rest of your software? If so, could we add a LICENSE file?
I run agents for a few different Buildkite orgs on my machine, and I'd like to use bk run
with the non-default one. To that end, I'd like to be able to do things like this without the env vars intended for the agent being overriden:
BUILDKITE_CONFIG_PATH=/etc/buildkite-agent/bt-buildkite-agent.cfg bk run
What is the proper option to get the initial install working? The current docs result in the below error:
ubuntu@ip-10-221-102-20 ~ $ bk configure
ββββ Ok! Let's get started with configuring bk π
ββββ Configuring Buildkite GraphQL credentials
Create a GraphQL token at https://buildkite.com/user/api-access-tokens/new. Make sure to tick the GraphQL scope at the bottom.
GraphQL Token: anthonyXXXXXion.com β
π¨ No directory provided for file keyring
Keyring-backend option mentions file but if I specify a file I get a confusing error:
22:24 $ bk configure --keyring-backend=$HOME/test
π¨ enum value must be one of file, got '/home/ubuntu/test'
I tried using this from a custom GitHub Action, but it seemed to stall. I tried running the same thing locally, and I have the same problem βΒ it just stalls on the pipeline upload:
$ docker run -it --rm -v "$(PWD):/app" bk run local
>>> Starting local agent π€
>>> Starting build π
>>> Executing initial command: buildkite-agent pipeline upload
^C
>>> Gracefully shutting down...
π¨ Initial command failed: context canceled
The Dockerfile I'm using is here:
https://gist.github.com/toolmantim/c731730b4b4b7989f8f045956346bcfe
Any hints on how I'd be able to debug?
Currently one of the step for bk configure
requires configuration of github cred within a browser:
ββββ Let's configure your github.com credentials π»
We need to authorize this app to access your repositories. This authorization is stored securely locally, buildkite.com never gets access to it.
When you press enter, your default browser will open and authenticate to github.com
How to do this step in the command line in a browserless/headless environment?
After authentication; bk init process failed at this step:
Checking for pipeline file: .buildkite/pipeline.yml β
Checking for git repository and remote: https://github.com/XXXXX/XXXX β
Enter passphrase to unlock /Users/antho/.buildkite/keyring/:
Checking for buildkite pipeline: Failed to find organization id for slug "XXXXX" β
First of all hugs β€οΈ
Secondly, wondering the definition of the 'interface' to BK is?
Our workflow tends to involve developers working at their local machines, pushing changes up to a remote repository and having remote bk agents process the pipelines.
Intially, was hoping that bk local run
would simulate the running of an agent but after a brew install
I received
exec: "buildkite-agent": executable file not found in $PATH
How do you envision this local run working?
Is there scope for including the buildkite-agent install step as part of the brew formulae?
Ideally I'd love to run bk local run
as part of a git commit hook on devs locals.
Is the executable just a by-product of needing a BK API client?
I'm currently using a cobra client with github.com/buildkite/go-buildkite/buildkite.
It performs extra stuff like emitting statsd metrics based on the results of the api calls.
Are you imagining people use this tool in their pipelines for similar purposes?
If you'd like a customer to do user stories (https://www.mountaingoatsoftware.com/articles/questioning-your-users) with let me know, I would love to help out.
Again, lots of love for BK β€οΈ
Thank you.
Hi
First, thank you for the great tool which allows to test our pipelines on local.
Running the tool installed from brew on osx, I encountered an issue that seems to be linked with the peculiar status of the platform.
When running the sample pipeline of the plugin: junit-annotate, I see the following error happening:
bk local run .buildkite/pipeline.integration_tests.yml
(...)
--- :junit: Processing the junits
docker: Error response from daemon: Mounts denied:
The path /var/folders/kf/fsdklxv57_v6kq5ydjxpj6g00000gp/T/buildkite-plugins-670558647/github-com-buildkite-plugins-junit-annotate-buildkite-plugin-v1-7-0/ruby
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.
.
ERRO[0000] error waiting for container: context canceled
π¨ Error: The command exited with status 125
Apparently, when trying to mount such a folder on a mac, one would need to prefix /var
with /private
and use instead /private/var
according to this comment
Is this a problem of the plugin or of the bk local agent?
I tried downloading and running the cli for a local build, but it just sat there doing nothing for 10 minutes until I figured it got stuck somehow and hit ^C:
karl@chrx:~/work/goprofile_test$ bk local run
>>> Starting local agent π€
>>> Starting build π
>>> Executing initial command: buildkite-agent pipeline upload
^C
>>> Gracefully shutting down...
π¨ Initial command failed: context canceled
When I ran it with debug, it complained about permission problems, and then got stuck again:
karl@chrx:~/work/goprofile_test$ bk local run --debug
2019/12/03 13:25:51 [keyring] Considering backends: [pass file]
2019/12/03 13:25:51 [keyring] Failed backend pass: The pass program is not available
2019/12/03 13:25:51 Serving API on http://127.0.0.1:35269
2019/12/03 13:25:51 Creating bootrap script at /tmp/bootstrap-458222797
>>> Starting local agent π€
>>> Starting build π
>>> Executing initial command: buildkite-agent pipeline upload
open /etc/buildkite-agent/buildkite-agent.cfg: permission denied^C
>>> Gracefully shutting down...
π¨ Initial command failed: context canceled
How do I fix this?
I'm attempting to use bk run local
to test my pipeline, but am having issues.
I ran this with --debug
and here are some relevant lines.
2019/01/15 15:59:08 [http] POST /jobs/dc7e0e82-3e6b-484e-a553-94a05bb8b1cf/chunks?sequence=1&offset=0&size=50 127.0.0.1:49280
2019-01-15 15:59:08 DEBUG local β³ POST http://127.0.0.1:37989/jobs/dc7e0e82-3e6b-484e-a553-94a05bb8b1cf/chunks?sequence=1&offset=0&size=50 (HTTP/1.1 200 OK 1.311835ms)
2019-01-15 15:59:08 DEBUG local [LogStreamer] Shutting down all workers
2019-01-15 15:59:08 DEBUG local [JobRunner] Waiting for all other routines to finish
2019-01-15 15:59:08 DEBUG local [JobRunner] Deleted env file: /tmp/job-env-dc7e0e82-3e6b-484e-a553-94a05bb8b1cf033464002
2019-01-15 15:59:08 DEBUG local [LogStreamer/Worker#1] Worker has shutdown
2019-01-15 15:59:08 DEBUG local [LogStreamer/Worker#0] Worker has shutdown
2019-01-15 15:59:08 DEBUG local [LogStreamer/Worker#2] Worker has shutdown
2019-01-15 15:59:08 DEBUG local PUT http://127.0.0.1:37989/jobs/dc7e0e82-3e6b-484e-a553-94a05bb8b1cf/finish
2019/01/15 15:59:08 [http] PUT /jobs/dc7e0e82-3e6b-484e-a553-94a05bb8b1cf/finish 127.0.0.1:49280
fork/exec /tmp/bootstrap-012338360: text file busy
2019-01-15 15:59:08 DEBUG local β³ PUT http://127.0.0.1:37989/jobs/dc7e0e82-3e6b-484e-a553-94a05bb8b1cf/finish (HTTP/1.1 200 OK 1.007612ms)
2019-01-15 15:59:08 DEBUG local Received signal `INT`
2019-01-15 15:59:08 INFO local Received CTRL-C, send again to forcefully kill the agent
2019-01-15 15:59:08 INFO local Gracefully stopping agent. Waiting for current job to finish before disconnecting...
2019-01-15 15:59:08 INFO local Finished job dc7e0e82-3e6b-484e-a553-94a05bb8b1cf
2019-01-15 15:59:08 INFO local Disconnecting local...
2019-01-15 15:59:08 DEBUG local POST http://127.0.0.1:37989/disconnect
2019/01/15 15:59:08 [http] POST /disconnect 127.0.0.1:49276
2019-01-15 15:59:08 DEBUG local β³ POST http://127.0.0.1:37989/disconnect (HTTP/1.1 200 OK 1.091944ms)
π¨ Initial command failed: Job failed with code 1
It almost looks like everything worked, but it had issues finalizing?
My pipeline:
steps:
- label: Example Test
command: echo "Hello"
Environment:
$ buildkite-agent --version
buildkite-agent version 3.8.2, build 2742
$ uname -a
Linux jerome-arch 4.20.0-1-MANJARO #1 SMP PREEMPT Mon Dec 24 08:20:48 UTC 2018 x86_64 GNU/Linux
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.