mdubourg001/glci

🦊 Test your Gitlab CI Pipelines changes locally using Docker.

glci's Issues

gitlab-ci-local

Hey, I couldn't find any sane way to reach you @mdubourg001, so I hope its ok i'm doing it like this
I just wanted to let you know, that i've been working on https://github.com/firecow/gitlab-ci-local for a while and it has support for artifacts, rules, shell-executor, docker-executor and doesn't leave root owned files in project directories

Suggestion: Consider using Gitlab's parser

You might have considered this already. If so, please ignore and close this issue.

I really like the idea of glci and applaud you for starting it.

When trying out the tool, I noticed that not all of the keywords of the Gitlab reference are supported (yet). [btw: Thanks for addressing my issues immediately.] Now, I wonder how much sense it makes to keep trying to play catch up with the official reference. It is open source after all. Would it make sense and would it be possible to rely on Gitlab's official implementation of the parser and continue from there?

There are several potential blockers (too tight coupling with their runner implementation might be one). And maybe it would take out the fun for you. However, just speaking from the perspective of a potential user of this tool, I think this approach is at least worth to be considered. It might allow to quickly get a larger feature set working.

Again, this is just a suggestion and I am not mad if you ignore it 😃

Running git commands fail

When running in GitLab CI the folder that you gets put in when running scripts is a Git repository. This means that for instance if you want to handle things like fetching submodules yourself, then you can do that. However, with glci it seems like that is not the case and the .git folder does not exist at all. This causes git commands such as git submodule update --init --recursive to fail.

The error, as expected is:

fatal: not a git repository (or any of the parent directories): .git

Can not log in to Docker registry

Running docker login -u <username> -p <password> will throw the following error:

Error: Cannot perform an interactive login from a non TTY device

I can use --password-stdin like:
echo "bar" | docker login --username foo --password-stdin
to get around this, but mainly pointing out that glci does not handle this the same way as GitLabs CI Runner would (at least not the Kubernetes ones).

Document requirement for libkrb5

I was experiencing this error when trying to install glci:

/bin/sh: 1: krb5-config: not found
gyp: Call to 'krb5-config gssapi --libs' returned exit status 127 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack

The fix for me was to apt install libkrb5-3 (on Ubuntu 20.04)

Cannot read property 'indexOf' of undefined

If I run glci locally on Windows, I get the following error:

TypeError: Cannot read property 'indexOf' of undefined
✘  - default

Local includes without / does not work

Example:

If I have a .gitlab-ci.yml file that starts with

include:
  - local: '.gitlab-ci-base-template.yml'

and that file is in the same folder as .gitlab-ci.yml, then glci tries to fetch it from the incorrect path. It seems like it expects the import to start with /.

This will lead to the following issue:

(node:70849) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '<redacted...>/<folder_with_gitlab-ci.yml>.gitlab-ci-base-template.yml'
    at Object.openSync (fs.js:476:3)
    at Object.readFileSync (fs.js:377:35)
    at main (<redacted...>/.config/yarn/global/node_modules/glci/dist/cli.min.js:206:44)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:70849) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:70849) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Say that I'm in /test and I want to import .base.yml then it will try to include /tests.base.yml.

Child pipelines?

I'm going to guess that this tool doesn't support child pipelines? When I try running a valid pipeline configuration that contains child pipelines, glci errors out with TypeError: commands is not iterable once it hits the job that triggers the child pipeline. Is this something you have on your roadmap?

Official Docker Image

It would be great if this had an official docker image from you. Here is what I was thinking for a run syntax:

docker run -it mdubourg001/glci --only-jobs # or whatever extra args are provided.

Here is what I think that the Dockerfile should look like:

# docker build -t glci:local . && docker run -it glci:local

# I couldn't find a better image that worked (apparently python's a dependency)
FROM okdocker/pynode

RUN yarn global add glci # maybe we want to install from the local version?

ENTRYPOINT ["/usr/local/bin/glci"]
CMD ["--help"]

Here's an example of me running a local version of it:

I'd be happy to help incorporate the CI portion of it if you want help.

Thanks for making some amazing tooling! This fills a need that I have had for some time, but was too lazy to try to tackle.

Extends with included, local yml files doesn't find job

When trying to use extends with a job defined in locally included yml file, glci 0.4.1 tells me, Can't extend job '.docker_build': job doesn't exist..

Simplified example:

docker.yml

.docker_build:
  image: docker:stable
  scripts:
    .

.gitlab-ci.yml

include:
  - local: '/gitlab-ci-templates/docker.yml'

my job:
  extends: .docker_build
  variables:
    ...

That example works on GitLab.com.

Is this a limitation of glci or a bug?

Split long commands syntax not supported

When running a task formatted as so:

pre:
  stage: pre-prepare
  script: |
    mkdir -p test1
    touch test1/test.txt

  artifacts:
    paths:
      - test1

it causes script to be read as a string and not an array.
I fixed it with adding this to the top of the execCommands function in index.

if(typeof commands === 'string' || commands instanceof String)
    commands = [commands];

I don't really have the time to submit a pull request right now or even manage it so I figured this would be a better option.

Running sourced script functions does not work

Say that you have the following file, scripts.sh with:

hello_world() {
    echo "Hello world!"
}

If I in the pipeline would be to do the following in my script or in my case before_script:

- source scripts.sh
- hello_word

Then I would expect "Hello world" to be echoed.

This is something that works as expected when running in a normal GitLab CI Runner, however in the case of glci I'm getting this:

sh: hello_world: not found

I'm guessing the the problem here is that you do something different from that the GitLab runner is doing. As it says sh here and not bash I do see why this is not working however. And you are running sh here it seems

glci/dist/cli.min.js

Line 126 in fad06ba

Cmd: ["sh", "-c", command],

So this is something that GitLab clearly does differently than glci.

Error when .gitlab-ci.yml doesn't have a stages section

GitLab by default has 3 stages: build, test and deploy (see https://docs.gitlab.com/ee/ci/yaml/#stages).

Thus, it is valid to have a .gitlab-ci.yml with no stages: defined, as long as each job is in one of those three stages. For instance, this yml that deploys my website is valid and works on GitLab.

However, when I run glci in that directory I get the error below:

(node:99545) UnhandledPromiseRejectionWarning: No 'stages' keyword found in your .gitlab-ci.yml.
(node:99545) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:99545) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I also get that error even if I run with the --only-jobs flag.

Install failing - something about gyp not supporting Python 3?

~ ❯ yarn global add glci                                                                                                                                                                                                               08:33:56
yarn global v1.22.10
[1/4] Resolving packages...
warning glci > nodegit > [email protected]: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future
warning glci > nodegit > node-gyp > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning glci > nodegit > node-gyp > request > [email protected]: this library is no longer supported
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /home/sam/.config/yarn/global/node_modules/nodegit: Command failed.
Exit code: 1
Command: node lifecycleScripts/preinstall && node lifecycleScripts/install
Arguments: 
Directory: /home/sam/.config/yarn/global/node_modules/nodegit
Output:
[nodegit] Running pre-install script
[nodegit] Running install script
node-pre-gyp
info it worked if it ends with ok
node-pre-gyp
info using [email protected]
node-pre-gyp info using
[email protected] | linux | x64
node-pre-gyp WARN
Using request for node-pre-gyp https download
node-pre-gyp
info check checked for "/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node" (not found)
node-pre-gyp
http GET https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz
node-pre-gyp http 404 https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz
node-pre-gyp WARN Tried to download(404): https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz 
node-pre-gyp
WARN Pre-built binaries not found for [email protected] and [email protected] (node-v88 ABI, glibc) (falling back to source compile with node-gyp) 
node-pre-gyp http 404 status code downloading tarball https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz
gyp

info it worked if it ends with ok
gyp
info using [email protected]
gyp info using [email protected] | linux | x64
gyp
info ok
gyp
info
it worked if it ends with ok
gyp
info using [email protected]
gyp info using [email protected] | linux | x64
gyp
ERR! configure error
gyp
ERR! stack Error: Python executable "/usr/bin/python" is v3.9.2, which is not supported by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
gyp ERR! stack     at PythonFinder.failPythonVersion (/home/sam/.config/yarn/global/node_modules/node-gyp/lib/configure.js:493:19)
gyp ERR! stack     at PythonFinder.<anonymous> (/home/sam/.config/yarn/global/node_modules/node-gyp/lib/configure.js:475:14)
gyp ERR! stack     at ChildProcess.exithandler (node:child_process:317:7)
gyp ERR! stack     at ChildProcess.emit (node:events:378:20)
gyp
ERR! stack     at maybeClose (node:internal/child_process:1067:16)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
gyp ERR!
System Linux 5.11.1-zen1-1-zen
gyp ERR! command "/usr/bin/node" "/home/sam/.config/yarn/global/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node" "--module_name=nodegit" "--module_path=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release" "--napi_version=7" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v88"
gyp ERR!
cwd /home/sam/.config/yarn/global/node_modules/nodegit
gyp ERR! node -v v15.10.0
gyp ERR! node-gyp -v v4.0.0
gyp ERR!
not ok
node-pre-gyp
ERR! build error
node-pre-gyp ERR!
stack Error: Failed to execute '/usr/bin/node /home/sam/.config/yarn/global/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node --module_name=nodegit --module_path=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v88' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/home/sam/.config/yarn/global/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack
at ChildProcess.emit (node:events:378:20)
node-pre-gyp ERR! stack     at maybeClose (node:internal/child_process:1067:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
node-pre-gyp
ERR! System Linux 5.11.1-zen1-1-zen
node-pre-gyp ERR! command "/usr/bin/node" "/home/sam/.config/yarn/global/node_modules/nodegit/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR!
cwd /home/sam/.config/yarn/global/node_modules/nodegit
node-pre-gyp ERR! node -v v15.10.0
node-pre-gyp ERR! node-pre-gyp -v v0.13.0
node-pre-gyp ERR! not ok
Failed to execute '/usr/bin/node /home/sam/.config/yarn/global/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node --module_name=nodegit --module_path=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v88' (1)
[nodegit] ERROR - Could not finish install
[nodegit] ERROR - finished with error code: 1
info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.

docker pull fails

My user does not have the docker group and I normally run docker with sudo. Might this be the cause of the crash?

---------------------------------------------------------------
Running job "build-image" for stage "build-1"
---------------------------------------------------------------

TypeError: Cannot read property 'indexOf' of undefined
    at Object.module.exports.parseRepositoryTag (/home/user/.config/yarn/global/node_modules/dockerode/lib/util.js:42:25)
    at Docker.pull (/home/user/.config/yarn/global/node_modules/dockerode/lib/docker.js:1411:23)
    at /home/user/.config/yarn/global/node_modules/glci/dist/cli.min.js:637:55
    at new Promise (<anonymous>)
    at main$1 (/home/user/.config/yarn/global/node_modules/glci/dist/cli.min.js:637:15)
✘  - build-image

Populate env variables from `variables:` if `default:` exists

Would be great to run the jobs with the variables defined with the variables: key. Right now this doesn't seem to work when default: is defined.

Great project btw!

Can't pull image from private registry

Hi, thanks for the project!
I'm running on macOS and I think I'm hitting an error pulling the Docker image for my CI. I've tried pulling the image manually but it didn't help. Here's the error I'm running into:

$ glci
---------------------------------------------------
Running job "golint" for stage "analysis"
---------------------------------------------------

/Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:383
  stream.on('data', onStreamEvent);
         ^

TypeError: Cannot read property 'on' of null
    at Modem.followProgress (/Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:383:10)
    at /Users/tobias/.config/yarn/global/node_modules/glci/dist/cli.min.js:324:24
    at /Users/tobias/.config/yarn/global/node_modules/dockerode/lib/docker.js:116:7
    at /Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:309:7
    at IncomingMessage.<anonymous> (/Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:328:9)
    at IncomingMessage.emit (node:events:390:22)
    at endReadableNT (node:internal/streams/readable:1307:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)

$ glci --version
0.1.0

The Docker image is hosted on a private repository, but I'm logged in locally.

OCI runtime exec failed

This same branch / commit ID builds successfully on my Gitlab server but when run locally with glci I get the following error.

$ glci --only-jobs=build
-----------------------------------------------
Running job "build" for stage "build"
-----------------------------------------------

ℹ - Using existing image "gcr.io/kaniko-project/executor:debug"
mkdir -p /root/.docker
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "process_linux.go:101: executing setns process caused \"exit status 1\"": unknown
✘  - build

The build stage in my .gitlab-ci.yml looks like this

build:
  stage: build
  image: gcr.io/kaniko-project/executor:debug
  before_script:
    - mkdir -p /root/.docker
    - echo $DOCKER_AUTH_CONFIG > /root/.docker/config.json
  script:
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $RUN_TAG --destination $BRANCH_TAG --destination $COMMIT_TAG
  tags:
    - kubernetes
  variables:
    GIT_SUBMODULE_STRATEGY: normal

v0.4.0 TypeError: Cannot read property 'untracked' of undefined

Just upgraded to v0.4.0 to test out the new --yml param you added (thank you, BTW), and I'm getting the following error when running my parent .gitlab-ci.yml file:

TypeError: Cannot read property 'untracked' of undefined
    at main$1 (/Users/gilzowp/.config/yarn/global/node_modules/glci/dist/cli.min.js:789:25)
    at processTicksAndRejections (node:internal/process/task_queues:94:5)

The gitlab-ci.yml file is fairly straightforward

stages:
  - setup
  - buildconfig
  - runbackstop

EnsureDeploy:
  stage: setup
  image:
    name: boyerj/platform-ci:0.6.2
  script:
    # redeploy the env to synthesize an env coming up after an MR. All code removed for local pipeline testing
   # simply state that the MR environment was deployed
    - echo "We just deployed the environment"
  
Find Environment:
  stage: setup
  image:
    name: boyerj/platform-ci:0.6.2
  script:
    # normally we would poll the environment to see if it has completed build and deploy events 
    #- ./scripts/digitalservice/cicd/platform-wait-for-build.sh
    # For local test, pretend we did and echo out
    - echo "The environment is now ready"
    # retrieves the ephemeral MR domains so we can test against them
    - php ./scripts/digitalservice/cicd/gitpip-save-urls.php
    - mkdir artifacts
    - cp /home/urls.csv artifacts/urls.csv
    - cp /home/urls.json artifacts/urls.json
  artifacts:
    paths:
      - artifacts/urls.csv
      - artifacts/urls.json

CheckArtifact:
  image:
    name: boyerj/platform-ci:0.6.2
  stage: buildconfig
  script:
    # Builds a child pipeline gitlab-ci.yml file, one job per domain to run visual regression tests
    - php ./scripts/digitalservice/cicd/buildBackstopPipeline.php
  artifacts:
    paths:
      - artifacts/backstop.gitlab-ci.yml
  dependencies:
    - Find Environment

RunOurBackStop:
  stage: runbackstop
  trigger:
    include:
      - artifact: artifacts/backstop.gitlab-ci.yml
        job: CheckArtifact

glci is failing and producing the mentioned error message immediately after the echo in the EnsureDeploy job.

Extending hidden jobs does not work

In my production CI configuration (which works in Gitlab), I ran into #19. The proposed solution there is to add a missing image.name. Images are defined in my CI definition, so I am trying to figure out what causes glci to not find this information. We use extends with hidden jobs from included YAML files.

When trying to distill a minimum example to reproduce the issue, I ran into the following problems.

First attempt

In my first attempt, I defined one template and one job directly in .gitlab-ci.yml:

.example-template:
    image:
        name: alpine

example-job:
    extends:
        - .example-template
    script:
        echo "Hello World"

This works in Gitlab but fails when running glci (but not with the error from #19 which I tried to reproduce):

> glci
(node:58265) UnhandledPromiseRejectionWarning: Error: could not find repository from '.'
(node:58265) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:58265) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I don't know what's going on here, so I switched to another approach.

Second attempt

Next, I tried splitting the files and including them in .gitlab-ci.yml (like I do in my production CI config):

.gitlab-ci.yml

include:
    - local: .gitlab/ci/template.gitlab-ci.yml
    - local: .gitlab/ci/example.gitlab-ci.yml

.gitlab/ci/template.gitlab-ci.yml

.example-template:
    image:
        name: alpine

.gitlab/ci/example.gitlab-ci.yml

example-job:
    extends:
        - .example-template
    script:
        echo "Hello World"

This also works in Gitlab but fails for yet another reason with glci:

> glci
----------------------------------------------------------
Running job ".example-template" for stage "test"
----------------------------------------------------------

ℹ - Pulling image "alpine"...
TypeError: commands is not iterable
✘  - .example-template

Here, it seems as if glci tries to run the hidden job instead of just using it as a template.

At this point my list of one issue has grown to three and I am stopping my attempts to reproduce the issue for the time being. I am happy to assist in further debugging though. glci seems like it has great potential for faster iterations on Gitlab CI configuration after all.

Docker in docker service support

Do you plan to add support for the docker:dind service ?
It currently seems to be ignored.
For this simple .gitlab-ci.yml:

stages:
  - build

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker run amazon/aws-cli

this the log that I get:

glci (v0.4.1)

   build   
-----------
|---------|
|  build  |
|---------|

-----------------------------------------------
Running job "build" for stage "build"
-----------------------------------------------

ℹ - Using existing image "docker:latest"
docker run amazon/aws-cli
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.

✘  - build

UnhandledPromiseRejectionWarning

Hello,
I was happy to find today this project. Unfortunately, it doesn't work for me. I just dist-upgraded to Ubuntu 20. glci writes these errors:

$ glci
glci (v0.4.3)

(node:3812) UnhandledPromiseRejectionWarning: TypeError: Cannot use 'in' operator to search for 'local' in p
    at main$1 (/home/b/.config/yarn/global/node_modules/glci/dist/cli.min.js:528:19)
(node:3812) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3812) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Is it a known issue ?
Regards.

`extends:` is not treated as array

Thank you for quickly adding the requested extends functionality. Unfortunately, it only works for a single template while extends: is an array.

To reproduce, try this .gitlab-ci.yml file:

.first-example-template:
    image:
        name: alpine

.second-example-template:
    only:
        refs:
            - master

example-job:
    extends:
        - .first-example-template
        - .second-example-template
    script:
        echo "Hello World"

It seems as if glci joins the names of the extended jobs instead of iterating over them:

> glci
Can't extend job '.first-example-template,.second-example-template': job doesn't exist.

Command 'glci' not found

When running "yarn global add glci" the following is return in the konsole:

However, when I try to run "glci" after this in the repository, I get the following message:

How can this be fixed?