firecow/gitlab-ci-local

Tired of pushing to test your .gitlab-ci.yml?

Tired of pushing to test your .gitlab-ci.yml?

Run gitlab pipelines locally as shell executor or docker executor.

Get rid of all those dev specific shell scripts and make files.

Table of contents

• Examples
  • docker-compose-nodejs
  • docker-swarm-php
  • docker-in-docker-build
• Installation
• Convenience
  • CLI options via shell
  • DotEnv file
  • Bash alias
  • Tab completion
  • Listing jobs
• Quirks
  • Tracked Files
  • Local Only
  • Home File Variables
  • Remote File Variables
  • Project File Variables
  • Decorators
  • Includes
  • Artifacts
• Development
  • Scripts
  • Package binaries

Installation

Linux based on Debian

Users of Debian-based distributions should prefer the the Deb822 format, installed with:

sudo wget -O /etc/apt/sources.list.d/gitlab-ci-local.sources https://gitlab-ci-local-ppa.firecow.dk/gitlab-ci-local.sources
sudo apt-get update
sudo apt-get install gitlab-ci-local

If your distribution does not support this, you can run these commands:

curl -s "https://gitlab-ci-local-ppa.firecow.dk/pubkey.gpg" | sudo apt-key add -
echo "deb https://gitlab-ci-local-ppa.firecow.dk ./" | sudo tee /etc/apt/sources.list.d/gitlab-ci-local.list
sudo apt-get update
sudo apt-get install gitlab-ci-local

Note that the path /etc/apt/sources.list.d/gitlab-ci-local.list is used in the file gitlab-ci-local.list. If you change it in these commands you must also change it in /etc/apt/sources.list.d/gitlab-ci-local.list.

NPM

npm install -g gitlab-ci-local

Macos

bash version must be above or equal 4.x.x

brew install gitlab-ci-local

Windows (Git bash)

• Install gitbash
• Install rsync

Download and put binary in C:\Program Files\Git\mingw64\bin

curl -L https://github.com/firecow/gitlab-ci-local/releases/latest/download/win.gz | gunzip -c > /c/Program\ Files/Git/mingw64/bin/gitlab-ci-local.exe

Executing gitlab-ci-local with --variable MSYS_NO_PATHCONV=1 can be useful in certain situations

Convenience

CLI options via shell

# Overrides .gitlab-ci.yml as the default git ci/cd file
export GCL_NEEDS='true' >> ~/.bashrc
export GCL_FILE='.gitlab-ci-local.yml' >> ~/.bashrc
export GCL_VARIABLE="IMAGE=someimage SOMEOTHERIMAGE=someotherimage"

DotEnv file

Add a .gitlab-ci-local-env file to the current working directory or a .env file in $HOME/.gitlab-ci-local

# Overrides .gitlab-ci.yml as the default git ci/cd file
FILE=doctor-strange.yml # --file

# Always runs needed jobs, when gitlab-ci-local <job-name> is called
NEEDS=true # --needs

All cli options can be assigned default values this way

Bash alias

echo "alias gcl='gitlab-ci-local'" >> ~/.bashrc

Tab completion

gitlab-ci-local --completion >> ~/.bashrc 

Logging options

export GCL_TIMESTAMPS=true # or --timestamps: show timestamps in logs
export GCL_MAX_JOB_NAME_PADDING=30 # or --maxJobNamePadding: limit padding around job name
export GCL_QUIET=true # or --quiet: Suppress all job output

List Pipeline Jobs

Sometimes there is the need of knowing which jobs will be added before actually executing the pipeline. GitLab CI Local is providing the ability of showing added jobs with the following cli flags.

--list

The command gitlab-ci-local --list will return pretty output and will also filter all jobs which are set to when: never.

name        description  stage   when        allow_failure  needs
test-job    Run Tests    test    on_success  false      
build-job                build   on_success  true           [test-job]

--list-all

Same as --list but will also print out jobs which are set to when: never (directly and implicit e.g. via rules).

name        description  stage   when        allow_failure  needs
test-job    Run Tests    test    on_success  false      
build-job                build   on_success  true           [test-job]
deploy-job               deploy  never       false          [build-job]

--list-csv

The command gitlab-ci-local --list-csv will output the pipeline jobs as csv formatted list and will also filter all jobs which are set to when: never. The description will always be wrapped in quotes (even if there is none) to prevent semicolons in the description disturb the csv structure.

name;description;stage;when;allow_failure;needs
test-job;"Run Tests";test;on_success;false;[]
build-job;"";build;on_success;true;[test-job]

--list-csv-all

Same as --list-csv-all but will also print out jobs which are set to when: never (directly and implicit e.g. via rules).

name;description;stage;when;allow_failure;needs
test-job;"Run Tests";test;on_success;false;[]
build-job;"";build;on_success;true;[test-job]
deploy-job;"";deploy;never;false;[build-job]

Quirks

git+http isn't properly supported #605 and has certain quirks

Tracked Files

Untracked and ignored files will not be synced inside isolated jobs, only tracked files are synced.

Remember git add

Local Only

local-only-job:
  rules:
    - { if: $GITLAB_CI == 'false' }

local-only-subsection:
  script:
    - if [ $GITLAB_CI == 'false' ]; then eslint . --fix; fi
    - eslint . 

Home file variables

Put a file like this in $HOME/.gitlab-ci-local/variables.yml

---
project:
  gitlab.com/test-group/test-project.git:
    # Will be type Variable and only available if remote is exact match
    AUTHORIZATION_PASSWORD: djwqiod910321
  gitlab.com:project/test-group/test-project.git: # another syntax
    AUTHORIZATION_PASSWORD: djwqiod910321

group:
  gitlab.com/test-group/:
    # Will be type Variable and only available for remotes that include group named 'test-group'
    DOCKER_LOGIN_PASSWORD: dij3213n123n12in3

global:
  # Will be type File, because value is a file path
  KNOWN_HOSTS: '~/.ssh/known_hosts'
  DEPLOY_ENV_SPECIFIC:
    type: variable # Optional and defaults to variable
    values:
      '*production*': 'Im production only value'
      'staging': 'Im staging only value'
  FILE_CONTENT_IN_VALUES:
    type: file
    values:
      '*': |
        Im staging only value
        I'm great for certs n' stuff

Variables will now appear in your jobs, if project or group matches git remote, globals are always present

Remote file variables

gitlab-ci-local --remote-variables [email protected]:firecow/example.git=gitlab-variables.yml=master

Project file variables

Put a file like this in $CWD/.gitlab-ci-local-variables.yml

---
AUTHORIZATION_PASSWORD: djwqiod910321
DOCKER_LOGIN_PASSWORD: dij3213n123n12in3
# Will be type File, because value is a file path
KNOWN_HOSTS: '~/.ssh/known_hosts'

Variables will now appear in your jobs.

Decorators

The @Description decorator

Adds descriptive text to gitlab-ci-local --list

# @Description Install npm packages
npm-install:
  image: node
  artifacts:
    paths:
      - node_modules/
  script:
    - npm install --no-audit

The @Interactive decorator

# @Interactive
interactive-shell:
  rules:
    - if: $GITLAB_CI == 'false'
      when: manual
  script:
    - docker run -it debian bash

The @InjectSSHAgent decorator

# @InjectSSHAgent
need-ssh:
  image: kroniak/ssh-client
  script:
    - ssh-add -L

The @NoArtifactsToSource decorator

Prevent artifacts from being copied to source folder

# @NoArtifactsToSource
produce:
  stage: build
  script: mkdir -p path/ && touch path/file1
  artifacts: { paths: [ path/ ] }

A global configuration is possible when setting the following flag

gitlab-ci-local --no-artifacts-to-source

Includes

Includes from external sources are only fetched once. Use --fetch-includes to invoke an external fetching routine.

Artifacts

Shell executor jobs copies artifacts to host/cwd directory. Use --shell-isolation option to mimic correct artifact handling for shell jobs.

Docker executor copies artifacts to and from .gitlab-ci-local/artifacts

Development

You need nodejs 18+

Scripts

# Install node_modules
npm install

# Compiled typescript to javascript
npm run build

# Run all tests
npm run test

# Run individual test-case
npx jest tests/test-cases/cache-paths-not-array

It's also possible to run individual .gitlab-ci.yml, via node src/index.js --cwd examples/docker-compose-nodejs

Package binaries

npm run pkg-linux
npm run pkg-win
npm run pkg-macos
npm run pkg-all

gitlab-ci-local's Issues

Support global `before_script`

Empty needs tag means, start right away

extends: only works with arrays

extends only works with array, not with string.

.base-demo:
    before_script: [echo "Demonstration"]
demo-extend-1:
    extends: [ .base-demo ]
    script: echo "This is '$CI_JOB_NAME' on GITLAB_CI='$GITLAB_CI'"
demo-extend-2:
    extends: [ demo-extend-1 ]
    variables:
        FOO: BAR
demo-extend-3:
    extends: demo-extend-1
    variables:
        FOO: BAZ

Expected output (and expected exit code: 0):

Demonstration
This is 'demo-extend-3' on GITLAB_CI='false'

Actual output (and actual exit code: 1):

d is used by demo-extend-3, but is unspecified

gitlab-ci-local version: 4.6.0

Postfix CI_PIPELINE_ID with -local

To avoid potential clashes with real CI_PIPELINE_ID's

--stage option, to run all jobs in a single stage

gitlab-ci-local --stage check

Shell-executor build folder isolation

First of all thanks for this great tool, it's awesome :)

I wish there would be a way to let the pipeline run in another isolated directory so that changes done by the pipeline don't affect the repository itself.

For example the command could look like this:

gitlab-ci-local --dir build

This would then create a directory called build, copy the project's content into it and run the pipeline in that directory.

Add package.json keywords

Verbosity on .gitlab-ci-local.yml overrides and appends.

Multiple job execution with needs

eg. gitlab-ci-local build deploy --needs

End report doesn't contain failed after_script's

Empty state file gives error

gitlab-ci-local
gitlab-ci-local [job]
Runs the entire pipeline or a single [job]
Positionals:
job Jobname to execute [string]
Options:
--help Show help [boolean]
--version Show version number [boolean]
--manual One or more manual jobs to run during a pipeline [array]
--list List jobs and job information [string]
--cwd Path to a gitlab-ci.yml [string]
--completion Generate bash completion script [string]
--needs Run needed jobs, when executing a single job [boolean]
TypeError: Cannot read property 'pipelineIid' of undefined
at Object.incrementPipelineIid (/snapshot/gitlab-ci-local/dist/predefined_variables.js)
at Object.handler (/snapshot/gitlab-ci-local/dist/default_cmd.js)
at Object.runCommand (/snapshot/gitlab-ci-local/node_modules/yargs/lib/command.js:240:40)
at Object.parseArgs [as _parseArgs] (/snapshot/gitlab-ci-local/node_modules/yargs/yargs.js:1182:35)
at Object.get [as argv] (/snapshot/gitlab-ci-local/node_modules/yargs/yargs.js:1088:21)
at Object. (/snapshot/gitlab-ci-local/dist/index.js)
at Module._compile (pkg/prelude/bootstrap.js:1320:22)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1176:10)
at Module.load (internal/modules/cjs/loader.js:1000:32)
at Function.Module._load (internal/modules/cjs/loader.js:899:14)

Error after run

pkg/prelude/bootstrap.js:1261
      return wrapper.apply(this.exports, args);
                     ^

TypeError: undefined is not iterable (cannot read property Symbol(Symbol.iterator))
    at new Parser (/snapshot/gitlab-pipeline-local/dist/parser.js)
    at Object.<anonymous> (/snapshot/gitlab-pipeline-local/dist/index.js)
    at Module._compile (pkg/prelude/bootstrap.js:1261:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:993:10)
    at Module.load (internal/modules/cjs/loader.js:813:32)
    at Function.Module._load (internal/modules/cjs/loader.js:725:14)
    at Function.Module.runMain (pkg/prelude/bootstrap.js:1316:12)
    at internal/main/run_main_module.js:17:11

Handle syntax error in `stages`

If stages is missing a properly formatted list the error is:

TypeError: undefined is not a function
at Parser.init (/snapshot/gitlab-ci-local/dist/parser.js)
at async Function.create (/snapshot/gitlab-ci-local/dist/parser.js)
at async Object.handler (/snapshot/gitlab-ci-local/dist/default_cmd.js)

>cat .gitlab-ci.yml
---
stages: lint

How to add environment variables?

Is it possible?

YAML anchors do not work

I get error from bash ("[object: command not found") when I try to use YAML anchors with gitlab-ci-local.

Steps/files to reproduce:

With .gitlab-ci.yml:

stages: [test]
main:
  stage: test
  script: echo Demonstration

and .gitlab-ci-local.yml:

.example: &test_anchor
  echo 'Example'
demo:
  stage: test
  script:
    - echo 'Hello'
    - *test_anchor
    - echo 'Goodbye'

Expected output when running gitlab-ci-local demo:

demo starting...
demo $ echo 'Hello'
demo > Hello
demo $ echo 'Example'
demo > Example
demo $ echo 'Goodbye'
demo > Goodbye
demo finished in 24 ms

Actual output:

demo starting...
demo $ echo 'Hello'
demo > Hello
demo $ [object Object]
demo > bash: line 2: [object: command not found
demo finished in 25 ms exited with code 127

YAML syntax is OK. I checked with python-yq
Output from yq -Y .demo .gitlab-ci-local.yml:

stage: test
script:

• echo 'Hello'
• echo 'Example'
• echo 'Goodbye'

extends: can be a string, we only support list

`include:remote:` field

bash completion

Support for bash completion of:
commmands eg. exec
tasks for exec

Rename project to gitlab-ci-local?

Does gitlab-ci-local sound better, and describe the project best ?

if you have a job name in the needs that does not exist. you get a confusing message and the unspecified jobname is unknown

include:local isn't recursive

Fail on unknown option

Gitlab-ci-local should fail if an unknown option is passe to the executable.
The case being that a misspell can cause eg. a --list call to actually start building

Package as NPX tool

It would be useful to have this tool in NPM to be able to run it with npx gitlab-ci-local

• Automatic updates would be handled by NPM
• No need to install binaries (out of free software spirit or security concerns)
• No traces left after running, unless specifically installed it with npm i -g gitlab-ci-local

To do that, you basically need to add a runnable (in terms of POSIX shells) JS script that imports the dist, modify your package.json and publish the code as an NPM library.

Here's how I would do it

1. touch cli.js

2. chmod +x cli.js

3. Edit cli.js

+ #!/usr/bin/env node
+ require('./dist/index.js')

4. Edit package.json

{
  "name": "gitlab-ci-local",
  "main": "index.js",
+  "bin": "cli.js", 
  "scripts": {
    "pkg-linux", // ...

5. npm publish

I would have done it myself, but as you're the author, you should probably have the rights to name the package gitlab-ci-local yourself.

Implement `coverage:` field

job1:
  script: rspec
  coverage: '/Code coverage: \d+\.\d+/'

Print coverage in end report if coverage field is used

Make needed jobs run as default, when executing single job

This change would require a major bump
Add --no-needs option, once needs is default

Interactive decorator

# @Interactive
# @Description Opens an interactive shell attached to the stack
interactive-shell:
  stage: .post
  rules:
    - if: $GITLAB_CI == 'false'
      when: manual
  script:
    - docker run -it -u $(id -u):$(id -g) -w /app/ -v "$PWD/:/app/" debian bash

Print environment links and names at the end of a job/pipeline run

When run in successful print environment links and info

Support for as many predefined variables as possible

https://docs.gitlab.com/ee/ci/variables/predefined_variables.html

needs: from same or future stages doesn't report error as gitlab does

Project/Group/Runner variables imported from file in users home dir

gitlab-ci-local --set-global-var SSH_PRIVATE_KEY < $(cat ~/.ssh/id_rsa)

yarn-install:
  stage: .pre
  script:
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - cd client
    - yarn install

This will definently be needed sooner or later.

--quiet option

With --quiet option only the end report it printed.

Preview option

CMD option --preview

It should output the entire "merged, extended, included and expanded" yaml to stdout

gitlab-ci-local list jobs functionality

End run report

Finish with a report on each pipeline run to conclude success or not.

`when: ( always | on_failure )` support

Multiline script support

echo:
  script:
    - >
      if [ $GITLAB_CI = 'true' ]; then
        echo "Hey"
      fi

If pretty broken atm. Failing with

echo > bash: line 4: syntax error near unexpected token `;'
echo > bash: line 4: `;echo GCL_MARKER=$?'

Support dynamic pipelines

Since GitLab 13.4, it's possible to write CI/CD pipelines which dynamically create sub-pipelines.

It would be great to have support for that.

Currently, the tool considers to be the "trigger" stance to be a malformed job and suggests to use "script" tag in it.

Implement Rules (only, except are slowly being deprecated)

needs with unspecified job

needs tag doesn't fail, if you write a jobname that doesn't exist

Make it possible to use spaces in job names

multiple extends as string in depth > 1

If we have multiple extends as string, we will get an error saying . is unspecified.

Description for job via #comment

When running --list it would be nice to be able to show a description of some sort

print docker cp timing, docker pull timing, docker run timing, for debugging slow pipelines.

Would be nice if we had a way to execute gitlab with -v or --verbose, to show more information on the execution.

exec --needs

Add an --needs option to exec to ease debugging job dependencies

`include:template:` field

DOCKER_AUTH_CONFIG special project/group/runner variable

With docker executor working, its only a matter of time, before people will need this
Here is some info
https://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry

Support `service:` field

extends:[] limited to single level

Support for extends is limited to a single level.

Steps to reproduce:

.base-demo:
    before_script: [echo "Demonstration"]
demo-extend-1:
    extends: [ .base-demo ]
    script: echo "This is '$CI_JOB_NAME' on GITLAB_CI='$GITLAB_CI'"
demo-extend-2:
    extends: [ demo-extend-1 ]
    variables:
        FOO: BAR

Expected output from "demo-extend-2":

Demonstration
This is 'demo-extend-2' on GITLAB_CI='false'

Actual output:

This is 'demo-extend-2' on GITLAB_CI='false'

Print include's being fetched in the beginning of the pipeline

Configure gitlab-ci-local per project via .gitlab-ci.local/config.yml