ktomk / pipelines Goto Github PK

View Code? Open in Web Editor NEW

107.0 7.0 12.0 3.88 MB

Pipelines - Run Bitbucket Pipelines Wherever They Dock

Home Page: https://ktomk.github.io/pipelines/

License: GNU Affero General Public License v3.0

PHP 88.83% Shell 8.80% Makefile 0.66% CSS 0.25% Python 1.18% HTML 0.27%

pipelines local-build docker bitbucket-pipelines pipeline-runner

pipelines's Issues

YAML Anchors

Would it be possible to have this work with YAML anchors? We use them extensively, and it would be nice to have this tool automatically expand them if found in the file as a pre-step, since it bombs like this when ancors are used:
branches/dev ERROR Missing required property 'step'

Services support (e.g. mysql)

Does this awesome tool support services like the real bitbucket pipeline does?

`step.clone-path` does not match current Bitbucket behavior

The default config parameter for step.clone-path mentioned in this doc is /app whereas when a pipeline runs in Bitbucket (as of 10/29/21) the mount point is /build. This discrepancy should be resolved so that users do not have to manually configure this option to mimic the full Bitbucket behavior

'image' required in service definition

This failure occurs when I have defined 'docker' as a service in order to increase the RAM size.

bitbucket-pipelines.yml

....
definitions: 
  services:
    docker:
      memory: 3072

....

Error

    container......: pipelines-1.Format-Validation.default.pts_objects
pipelines: file parse error: 'image' required in service definition

PHP Deprecated: trim(): ... no pipeline to run!

Just installed pipelines today and I'm brand new to this util. I've installed it using phive on macOS and setup my pipelines executable globally. I do have a bitbucket-pipelines.yml file setup and configured correctly. When just running the pipelines command with no additional flags in the same folder as the yaml file, it fails showing this error:

PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
PHP Deprecated:  trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49

Deprecated: trim(): Passing null to parameter #1 ($string) of type string is deprecated in phar:///Users/user/.phive/phars/pipelines/vendor/ktomk/symfony-yaml/Inline.php on line 49
pipelines: no pipeline to run!

🙋🏻‍♂️How do I fix this deprecation issue?

PHP Info

php --version
PHP 8.1.8 (cli) (built: Jul  8 2022 12:51:36) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.8, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.8, Copyright (c), by Zend Technologies

Dockerfile for pipelines?

Hi there,

Is there a Dockerfile with the pipelines command and PHP already installed? The use for this would be to allow users to run bitbucket pipelines through docker using your tool. Something like:

docker run -v "path/to/docker.sock:..." <pipelines-image-tag> [PIPELINES OPTIONS] <PIPELINES ARGS>

If not, have you given any thought to whether this is possible? My team is looking for a way to run BBP locally, and this tool looks promising. I might be able to contribute a Dockerfile like this if we end up going that way.

pipelines: file parse error: 'image' required in service definition

The problem occurs when we define a custom service
Ex:

definitions:
   services:
     docker:
       memory: 5120
     docker-7g:
       type: docker
       memory: 7168

Use official JSON schema for YML configuration file

Hello @ktomk,

I found a link to this tool in your reply to a Atlassian Community question. I just wanted to let you know that there's now an official JSON schema for Pipelines YML configuration file format available at the following URL: https://api.bitbucket.org/schemas/pipelines-configuration

I imagine it might be easier for you to bundle it into your tool cause you won't need to worry about keeping the schema up to date with the new Pipelines features.

Let me know if you have any questions.

Cheers,
Daniil

cannot parse yaml files with dash as indentation

Yaml spec 1.2 says:

The “-”, “?” and “:” characters used to denote block collection entries are perceived by people to be part of the indentation. This is handled on a case-by-case basis by the relevant productions.

However piplines fails when the file uses dash as identitation as follows:

pipelines:
  default:
  - step:
      name: Build and Test

it tells, theres no steps defined for default pipeline

More Docker Client Versions

Hi @ktomk!

I would like to say I've using your tool to a great degree and really appreciate your supporting it!

I wanted to ask if we could get more versions of docker? Right now the supported versions are

docker-17.12.0-ce-linux-static-x86_64
docker-18.09.1-linux-static-x86_64
docker-19.03.1-linux-static-x86_64

according to pipelines --docker-client-pkgs. Some scripts require more recent flags, such as --all-tags of docker push. I have used this flag successfully in bitbucket pipelines and therefore think some newer versions of the docker client would be helpful. I am not sure exactly which version is used in bitbucket.

Also I would like to state that I am using pipelines version 0.0.68 which appears to be the most recent tag.

Dealing with SSH stuff

It would be great to have a way to copy SSH stuff (public/private keys and host key verification) into the container. This way we could copy the same SSH keys used by the real pipeline.

What do you think?

Working with private repos

Can't make it work with a private repository

Failed to clone the [email protected]:.....git repository, try running in interactive mode so that you can enter your Bitbucket OAuth consumer credentials

Is there a way to share my local ssh credentials with the docker container?

pipelines: not a readable file:bitbucket-pipelines.yml

Hello,
This is some great work!
But I am stuck at the first step, while running the YAML file. it says it's not a readable file. My YAML file uses a scala-sbt image with sbt commands in the script.
Could you please help me with this issue I am facing. Do I do some things differently?

Thanks,
Prathamesh

BITBUCKET_REPO_SLUG holds the `.git` extension

In the bitbucket pipelines the BITBUCKET_REPO_SLUG does not hold the .git extension.

It's an inconsistency, but it's not documented anywhere by atlassian that is should not or cannot hold the .git extension.

Mounting Local Volumes to allow application to access local files

Hello Sir,
I have been using your pipeline package to run my yaml.file
My application needs to access certain files on the local machine, which it cannot do unless a volume has been mounted.
I can see some functionality --mount in your readme, but I am not sure how to use it to mount certain volumes to allow my application to interact with it.
Could you please help me with this. I dont want to keep every file in the application working directory.

Thank you.

Validator fails on pipes in steps

Background

I'm trying to validate my pipelines with pipelines --show bitbucket-pipeline.yaml.

Using latest .phar release as of time of writing.

bitbucket-pipelines.yml:

pipelines:

  branches:
    develop:
    - step:
        name: Deploy to staging
        deployment: staging
        script:
          - pipe: atlassian/rsync-deploy:0.4.2
            variables:
              USER: 'ec2-user'
              SERVER: '${SERVER_HOST}'
              REMOTE_PATH: '/home/ec2-user/app'
              LOCAL_PATH: '$PWD'
              EXTRA_ARGS: '--exclude=".git*"'
          - 'echo "Hello, World!"'

Expected Results

Expected this file to pass with flying colours - verified its correctness with the official online validator.

Actual Results

pipelines --show bitbucket-pipelines.yml 
PIPELINE ID         IMAGES    STEPS                                                         
branches/develop    ERROR     'script' requires a list of commands, step #0 is not a command

(Some) Anchors Do Not Work - Raises Invalid YAML

Thanks for the awesome tool, it has/will save me a lot of time. I was running this on a new pipeline like

cd <my_project>
pipelines

and it says

bitbucket-pipelines.yml; verify the file contains valid YAML

But I can parse the file with python

from yaml import safe_load
with open( 'bitbucket-pipelines.yml', 'r' )  as file : safe_load( file )

and it passes in the bitbucket validator. When I run the pipeline on bitbucket it works. I can provide my yaml, but it will require some amendments as this is internal to my organization and thus I am hesitant. If I figure out what is making the parser upset I will add that to my post here.

It also might be worth considering that I am using the windows subsystem for linux.

Thank you again for any time, it is really appreciated.

EDIT : Debug output

Running pipelines --debug gets

pipelines: file parse error: YAML error: /mnt/c/MVE/.../bitbucket-pipelines.yml; verify the file contains valid YAML
pipelines: version 0.0.62-composer w/ php 7.4.3 (libyaml: n/a)
--------
class....: Ktomk\Pipelines\File\ParseException
message..: file parse error: YAML error: /mnt/c/.../<my_project>/bitbucket-pipelines.yml; verify the file contains valid YAML
code.....: 2
file.....: /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/File/File.php
line.....: 53
backtrace:
#0 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/App.php(135): Ktomk\Pipelines\File\File::createFromFile()
#1 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/ExceptionHandler.php(50): Ktomk\Pipelines\Utility\App->run()
#2 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/ExceptionHandler.php(65): Ktomk\Pipelines\Utility\ExceptionHandler->handle()
#3 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/App.php(90): Ktomk\Pipelines\Utility\ExceptionHandler->handleStatus()
#4 /home/adr1an/.config/composer/vendor/ktomk/pipelines/bin/pipelines(37): Ktomk\Pipelines\Utility\App->main()
#5 /home/adr1an/.config/composer/vendor/bin/pipelines(112): include('/home/adr1an/.c...')
#6 {main}
--------

EDIT

It turns out that the parser does not like anchors and this is the root of the problem.

/bin/sh: 5: source: not found

Our images in the bitbucket-pipelines have bash as default runner. Can this be adhered?

This does not only affect the source command, which we could easily replace by .. But also environment variables that should be present are not there.

coverage-checker.php deprecated notice

pipelines/lib/build/coverage-checker.php

Line 53 in 3e67dce

$xml->xpath(".//metrics/@${s}elements")

PHP Deprecated: Using ${var} in strings is deprecated, use {$var} instead in .../coverage-checker.php on line 53

needs to be either
$xml->xpath('.//metrics/@' . $s . 'elements')
or
$xml->xpath(".//metrics/@{$s}elements")

Pipelines Fatal Environment Definition Error is puzzling

Hi @ktomk! Sorry if I have raised too many issues. Hopefully this is not something caused by my configuration as it runs in bitbucket. I run pipelines in a repository with the following bitbucket configuration :

image: python:3.10

options :
  docker : true

definitions :
  services :
    mysql :
      variables :
        MYSQL_RANDOM_ROOT_PASSWORD : 'yes'
        MYSQL_DATABASE : 'mve_brain_sqlalchemy_tests'
        MYSQL_USER : 'adrian'
        MYSQL_PASSWORD : 'somepassword'
      image : mysql
  steps :
    - step : &create_env_test
        name : Create .env.test for testing since this file should never be commited.
        script :
          - export ENV_TEST_PATH=".env.test"
          - touch $ENV_TEST_PATH
          - echo BRAIN_MYSQL='{"database":"mve_brain_sqlalchemy_tests","host":"localhost","port":3306,"username":"adrian","password":"somepassword","drivername":"mysql+asyncmy"}' >> $ENV_TEST_PATH 
          - echo BRAIN_AUTH0='{"client_id":"$BRAIN_AUTH0__CLIENT_ID","client_secret":"$BRAIN_AUTH0__CLIENT_SECRET","secret_key":"$BRAIN_AUTH0__SECRET_KEY","audience":"http://localhost:8000/","issuer":"$BRAIN_AUTH0__ISSUER"}' >> $ENV_TEST_PATH
          - echo BRAIN_UVICORN='{"host":"0.0.0.0","port":8000,"reload":true}' >> $ENV_TEST_PATH
          - echo BRAIN_AUTHDUMMY='{"secret_key":"$BRAIN_AUTH_DUMMY_SECRET_KEY","client_secret":"$BRAIN_AUTH_DUMMY__CLIENT_SECRET","admin_secret":"$BRAIN_AUTH_DUMMY__ADMIN_SECRET","token_timeout":3600}' >> $ENV_TEST_PATH
          - echo BRAIN_API_INCLUDE_UNSAFE_ROUTES=1 >> $ENV_TEST_PATH
        artifacts :
          - .env.test

    - step : &invoke_tests
        name : Run tests for non-fetchers items.
        caches :
          - pip
        script :
          - pip install -r requirements.txt 
          - pip install -r requirements.dev.txt
          - BRAIN_API_PURPOSE=test python -m pytest
        services :
          - mysql
 
    - step : &build_containers
        name : Building prod and fetchers docker images (without an enironment files)
        caches :
          - pip
        script :
          - export ACR_URI="$ACR_NAME.azurecr.io"
          - export IMAGE_API="$ACR_URI/api:$BITBUCKET_COMMIT" 
          - export IMAGE_FETCHERS="$ACR_URI/fetchers:$BITBUCKET_COMMIT"
          - docker login $ACR_URI --username $BBSP_USERNAME --password $BBSP_PASSWORD

          - docker build -t "$IMAGE_API" -f "Dockerfile.prod" --target "prod" .
          - docker build -t "$IMAGE_FETCHERS" -f "Dockerfile.prod" --target "fetcher_runner" . 
          
          - docker run --name test1 --detach "$IMAGE_API" 
          - sleep 30 
          - docker stop test1
                    
          - docker push $IMAGE_API 
          - docker push $IMAGE_FETCHERS

    - step : &typecheck_source
        name : See if code passes mypy's type checking.
        caches :
          - pip
        script :
          - python -m pip install mypy sqlalchemy[mypy]
          - if ( python -m mypy . > results_mypy 2>&1 ); then echo 1; else echo 0; fi
        artifacts :
          - results_mypy

    - step : &lint_source
        name : See if code passes flake8. Try to autolint.
        caches :
          - pip 
        script :
          - pip install flake8 
          - if ( python -m flake8 . > results_flake8 2>&1 ); then echo 1; else echo 0; fi
        artifacts :
          - results_flake8

    - step  : &scan_source
        name : Check for secrets etc.
        caches : 
          - pip
        script :
          - pip install bandit
          - if ( bandit . > results_bandit 2>&1 ); then echo 1; else echo 0; fi
        artifacts :
          - results_bandit

  basic : &basic
    - step : 
        <<: *create_env_test
    - step :
        <<: *invoke_tests

  everything : &everything

    - parallel :
      - step : 
          <<: *scan_source
      - step :
          <<: *lint_source
      - step :
          <<: *typecheck_source
    - step : 
        <<: *create_env_test
    - step :
        <<: *invoke_tests

  everythingandbuild  : &everythingandbuild

    - parallel :
      - step : 
          <<: *scan_source
      - step :
          <<: *lint_source
      - step :
          <<: *typecheck_source
    - step : 
        <<: *create_env_test
    - step :
        <<: *invoke_tests
    - step :
        <<: *build_containers

pipelines:
  default : *basic
  branches :
    master : *everythingandbuild
    refactoring :  *everythingandbuild

  pull-requests : 
    '**' : *everything

But I get the following issue when I run 'pipelines':

pipelines: fatal: Variable definition error: '  "database" : "mve_brain_sqlalchemy_tests",'

I believe this to be a yaml parsing error, due to the dictionary/json like structure. I pushed my changes to bitbucket and everything worked correctly. I am using pipelines 0.0.35 ( EDIT: THIS IS WRONG, I AM USING 0.65 ) and installed it onto WSL2 ubuntu using composer version 2.3.4.

Thanks again for your help @ktomk! This project is very helpful to my productivity.

Failed to parse validated YAML file

The following simple bitbucket-pipelines.yml is rejected with "verify the file contains valid YAML: Unable to parse at line 2 (near "sleep 10;")."

.test-sleep: &test-sleep
  sleep 10;
  sleep 20;
  
.test-step: &test-sleep-step
  step:
    name: Simple Test
    script:
      - date;
      - *test-sleep
      - date;

pipelines:
  default:
    - parallel:
        - <<: *test-sleep-step

This file was validated by https://yamlchecker.com/ and https://bitbucket-pipelines.atlassian.io/validator and executes properly in Atlassian Bitbucket Pipelines.

bitbucket-pipelines.zip

Default pipeline with parallel fails validation

Hi thanks for an awesome utility

I've a problem when defining a default pipeline with parallel step it just stops with

$ pipelines --debug
pipelines: error: pipeline id 'default'
pipelines: file parse error: Missing required property 'step'
--------
class....: Ktomk\Pipelines\File\ParseException
message..: file parse error: Missing required property 'step'
code.....: 2
file.....: phar:///~/bin/pipelines/src/File/ParseException.php
line.....: 27
backtrace:
#0 phar:///~/bin/pipelines/src/File/Pipeline.php(106): Ktomk\Pipelines\File\ParseException::__()
#1 phar:///~/bin/pipelines/src/File/Pipeline.php(91): Ktomk\Pipelines\File\Pipeline->step()
#2 phar:///~/bin/pipelines/src/File/Pipeline.php(33): Ktomk\Pipelines\File\Pipeline->parseSteps()
#3 phar:///~/bin/pipelines/src/File/File.php(231): Ktomk\Pipelines\File\Pipeline->__construct()
#4 phar:///~/bin/pipelines/src/Utility/App.php(438): Ktomk\Pipelines\File\File->getById()
#5 phar:///~/bin/pipelines/src/Utility/App.php(141): Ktomk\Pipelines\Utility\App->getRunPipeline()
#6 phar:///~/bin/pipelines/src/Utility/ExceptionHandler.php(48): Ktomk\Pipelines\Utility\App->run()
#7 phar:///~/bin/pipelines/src/Utility/App.php(86): Ktomk\Pipelines\Utility\ExceptionHandler->handle()
#8 phar:///~/bin/pipelines/bin/pipelines(26): Ktomk\Pipelines\Utility\App->main()
#9 /~/bin/pipelines(14): require('phar:///~/...')
#10 {main}
--------

This is the smallest config I can reproduce the error with

bitbucket-pipelines.yml

image: atlassian/default-image:latest

pipelines:
  default:
    - step:
        script:
          - echo "Step 1"
    - parallel:
        - step:
            script:
              - echo "Step 2.1"
        - step:
            script:
              - echo "Step 2.2"

Does this tool support pipes?

I am getting this weird output:

+ echo "pipe: atlassian/git-secrets-scan:1.4.0 (pending feature)" # pipe feature is pending
printf %s '  FILES_IGNORED (**/node_modules): '; printf '%s ' **/node_modules; printf '\n' 

pipe: atlassian/git-secrets-scan:1.4.0 (pending feature)
  FILES_IGNORED (**/node_modules): **/node_modules

Support for `condition`

What I can see the library don't support condition and changesets

I saw that you have a test yaml file with the structure test/data/yml/condition.yml but it don't seem to be used for anything else than to verify that its a valid pipeline config

So is this something that would be possible to implement or would you be open to a pull-request?

Yes we use git. And we only use it in BitBucket

How the detection work in BitBucket:

In the branches sections they looks in the last commit and sees if any files matches the includePaths and if so run the step. This should be fairly simple to implement
In the pull-requests section it will look in all commits for changes that matches includePaths. This might be harder but maybe we could require an option with a destionation branch and do the check against that diff?

Usage example:

pipelines:
  pull-requests:
      '**': #this runs as default for any branch not elsewhere defined
          - step:
                script:
                    - phpunit
                condition:
                    changesets:
                        includePaths:
                            - src/
                            - composer.lock

          - step:
                script:
                    - eslint
                condition:
                    changesets:
                        includePaths:
                            - js/

    branches:
        master:
            - step:
                  script:
                      - phpunit
                  condition:
                      changesets:
                          includePaths:
                              - src/
                              - composer.lock

Support for 'docker' in 'options'

Hi again @ktomk. Sorry for raising another issue. I found that the options section does nothing for my pipelines, e.g.

image : node:16
options :
  docker : True
...

See for instance the YAMLs in in #14. IDK if this is a big ask; my experiences make me imagine it might require some docker magic. I think I could do it by using a different image than the one I am using, but it is easier to just have it in options.

Parallel step support for fail-fast and steps properties

When a parallel step is defined with the fail-fast and steps properties pipelines shows it as invalid.

Here is a sample that is perfectly valid and runs in BitBucket:

pipelines:
  custom:
    'custom-pipeline':
      - step: *stepA
      - parallel:
          fail-fast: true
          steps:
            - step: *stepB
            - step: *stepC

Pipelines does not allow the above and expects parallel steps to be defined only as an array, like this:

pipelines:
  custom:
    'custom-pipeline':
      - step: *stepA
      - parallel:
          - step: *stepB
          - step: *stepC

script non-zero exit status: 243

What are you trying to do?
Just want to execute bitbucket-pipelines.yml on my local machine before pushing the code to the repo.
So what problem are you facing?
In execution, everything is going well until I trigger the script.
You mean, you are able to successfully complete all operations without triggering the script?
Yes, if I don't trigger the script the "pipelines" complete execution successfully.
Okay fine, show me your bitbucket-pipelines.yml file.

image: node:16.15.1
pipelines:
    default:
      - step:
          name: test
          caches:
            - node
          script:
            - apt-get update
            - apt-get install postgresql-client -y
            - cd functions
            - npm install
            - sh create-db.sh test
            - sh run_migration.sh test
            - echo "Completed"
          services:
            - postgres
definitions:
  services:
    postgres:
      image: postgres:latest
      variables:
        POSTGRES_USER: 'root'
        POSTGRES_PASSWORD: 'root'

Cool, let me show the execution part.

rentsher@rentsher-ThinkPad-P14s-Gen-1:/work/napses/backend-core$ pwd
/home/rentsher/work/napses/backend-core
rentsher@rentsher-ThinkPad-P14s-Gen-1:/work/napses/backend-core$ ls
bitbucket-pipelines.yml Dockerfile functions package-lock.json pipe.sh README.md start_api.sh
rentsher@rentsher-ThinkPad-P14s-Gen-1:~/work/napses/backend-core$ pipelines
+++ step #1

name...........: "test"
effective-image: node:16.15.1
container......: pipelines-1.test.default.backend-core
container-id...: 923bded7d99b

+++ copying files into container...

+++ populating caches...

node node_modules (hit)

apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 Packages [7911 kB]
Get:5 http://security.debian.org/debian-security buster/updates/main amd64 Packages [336 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [8788 B]
Fetched 8494 kB in 3s (3241 kB/s)
Reading package lists...
apt-get install postgresql-client -y
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
distro-info-data lsb-release postgresql-client-11 postgresql-client-common
Suggested packages:
lsb postgresql-11 postgresql-doc-11
The following NEW packages will be installed:
distro-info-data lsb-release postgresql-client postgresql-client-11
postgresql-client-common
0 upgraded, 5 newly installed, 0 to remove and 15 not upgraded.
Need to get 1594 kB of archives.
After this operation, 6613 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 distro-info-data all 0.41+deb10u4 [6880 B]
Get:2 http://security.debian.org/debian-security buster/updates/main amd64 postgresql-client-11 amd64 11.16-0+deb10u1 [1413 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 lsb-release all 10.2019051400 [27.5 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 postgresql-client-common all 200+deb10u4 [85.1 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 postgresql-client all 11+200+deb10u4 [61.1 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 1594 kB in 0s (6349 kB/s)
Selecting previously unselected package distro-info-data.
(Reading database ... 23988 files and directories currently installed.)
Preparing to unpack .../distro-info-data_0.41+deb10u4_all.deb ...
Unpacking distro-info-data (0.41+deb10u4) ...
Selecting previously unselected package lsb-release.
Preparing to unpack .../lsb-release_10.2019051400_all.deb ...
Unpacking lsb-release (10.2019051400) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_200+deb10u4_all.deb ...
Unpacking postgresql-client-common (200+deb10u4) ...
Selecting previously unselected package postgresql-client-11.
Preparing to unpack .../postgresql-client-11_11.16-0+deb10u1_amd64.deb ...
Unpacking postgresql-client-11 (11.16-0+deb10u1) ...
Selecting previously unselected package postgresql-client.
Preparing to unpack .../postgresql-client_11+200+deb10u4_all.deb ...
Unpacking postgresql-client (11+200+deb10u4) ...
Setting up postgresql-client-common (200+deb10u4) ...
Setting up distro-info-data (0.41+deb10u4) ...
Setting up postgresql-client-11 (11.16-0+deb10u1) ...
update-alternatives: using /usr/share/postgresql/11/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up lsb-release (10.2019051400) ...
Setting up postgresql-client (11+200+deb10u4) ...
cd functions
npm install

up to date, audited 922 packages in 1s

89 packages are looking for funding
run npm fund for details

found 0 vulnerabilities

sh create-db.sh test
[ -z test ]
echo Creating db...
NODE_ENV=test npx env-cmd -f ./env/.env.test npx sequelize-cli db:create --env=test
Creating db...

script non-zero exit status: 243
rentsher@rentsher-ThinkPad-P14s-Gen-1:~/work/napses/backend-core$