batect/batect

(NOT MAINTAINED) Build And Testing Environments as Code Tool

⚠️ Batect is no longer maintained. More details

---

Batect allows you to define your development tasks (building, running, testing, linting and more) in terms of one or more Docker containers, run those tasks quickly and consistently everywhere, and easily share them with your team.

Check out batect.dev for documentation, tutorials, sample projects and news.

Documentation

All documentation is available on the website. Highlights include:

• Introduction
• Getting started tutorial
• Configuration file reference
• Sample projects
• Comparison with other tools

Videos

• Dockerised local build and testing environments made easy at Container Camp AU (July 2019): video

  Also presented at DevOpsDays Auckland (October 2019), DDD Sydney (September 2019) and DDD Melbourne (August 2019).

• Build & Testing Environments as Code: Because Life's Too Short Not To at Evolution by ThoughtWorks (June 2018): video, slides

Support and community

Batect uses GitHub Discussions for community support and Q&A.

Feedback

Please open an issue on GitHub if you run into a problem or have a suggestion.

You can see what new features and improvements are planned in the roadmap.

Contributing

See the contribution guide.

Acknowledgements

Thank you to the following people for their bug reports, pull requests, suggestions and feedback, in alphabetical order:

@Abhisha1, @aidansteele, @akamanocha, @alexswilliams, @alpha-er, @andeemarks, @asharma8438, @askfor, @assafw, @b-a-byte, @BethanyDrake-x, @Bidaya0, @binkley, @boxleytw, @Byron-TW, @camjackson, @carloslimasis, @catacgc, @cazgp, @chandantp, @chinwobble, @csxero, @da4089, @damian-bisignano, @DamianBis, @dan-neumegen-xero, @DavidHe1127, @dflook, @diachedelic, @DiegoAlpizar, @diffidentDude, @diwang-xero, @doug-ferris-mondo, @eichelkrauta, @ekamara, @erMaurone, @flo-everett-xero, @frglrock, @fwilhe2, @gabrielsadaka, @GerardWorks, @GoodDingo, @heyheman11, @hongyuanlei, @hpcsc, @hussein-joe, @ineffyble, @isaac-patterson, @jagregory, @jbduncan, @jmewes, @jobasiimwe, @kbalston, @Letitia-May, @mario-prabowo-xero, @marty-macfly, @mdlnr, @MichaelKnightXero, @minnn-minnn, @mjstrasser, @Mknight492, @Mubashwer, @mylesmacrae, @nashvan, @nesl247, @nkrul, @or1can, @pameck, @paulvalla-zorro, @priorax, @ryanb6920, @safiranugroho, @Sami5, @smozely, @SongGithub, @squirmy, @sschuberth, @svishal9, @TassSinclair, @thirkcircus, @Tobytee17, @Tzrlk, @wandrewni, @wilvk, @wyvern8, @xdavidnguyen, @yoyo-i3, @yoyo-ohno, @yuzhanglong, @ZhuYeXero, @zizhongzhang, and everyone else who has used the tool and provided feedback offline

Thank you to YourKit for providing a complimentary copy of the YourKit profiler.

And thank you to Thoughtworks for providing me a day of paid leave each month to work on Batect. If you'd like to join us at Thoughtworks, check out our current job openings.

batect's Issues

Build failed: Java 11 perhaps

batect.integrationtests.DockerClientIntegrationTest > describe a Docker client FAILED
    org.mockito.exceptions.base.MockitoException: 
    Mockito cannot mock this class: class batect.logging.Logger.
    Can not mock final classes with the following settings :
     - explicit serialization (e.g. withSettings().serializable())
     - extra interfaces (e.g. withSettings().extraInterfaces(...))

    You are seeing this disclaimer because Mockito is configured to create inlined mocks.
    You can learn about inline mocks and their limitations under item #39 of the Mockito class javadoc.

    Underlying exception : org.mockito.exceptions.base.MockitoException: Could not modify all classes [class java.lang.Object, class batect.logging.Logger]
        at batect.integrationtests.DockerClientIntegrationTest$1$1.invoke(DockerClientIntegrationTest.kt:334)
        at batect.integrationtests.DockerClientIntegrationTest$1$1.invoke(DockerClientIntegrationTest.kt:68)
        at org.jetbrains.spek.engine.SpekTestEngine$Collector.group(SpekTestEngine.kt:152)
        at org.jetbrains.spek.api.dsl.SpecBody$DefaultImpls.group$default(SpecBody.kt:13)
        at org.jetbrains.spek.api.dsl.StandardKt.describe(Standard.kt:10)
        at batect.integrationtests.DockerClientIntegrationTest$1.invoke(DockerClientIntegrationTest.kt:69)
        at batect.integrationtests.DockerClientIntegrationTest$1.invoke(DockerClientIntegrationTest.kt:68)

        Caused by:
        org.mockito.exceptions.base.MockitoException: Could not modify all classes [class java.lang.Object, class batect.logging.Logger]
            at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:137)
            at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:344)
            at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:159)
            at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:353)
            ... 7 more

            Caused by:
            java.lang.IllegalStateException: 
            Byte Buddy could not instrument all classes within the mock's type hierarchy

            This problem should never occur for javac-compiled classes. This problem has been observed for classes that are:
             - Compiled by older versions of scalac
             - Classes that are part of the Android distribution
                at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.triggerRetransformation(InlineBytecodeGenerator.java:122)
                at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.mockClass(InlineBytecodeGenerator.java:99)
                at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:37)
                at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator$1.call(TypeCachingBytecodeGenerator.java:34)
                at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:137)
                at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:344)
                at net.bytebuddy.TypeCache.findOrInsert(TypeCache.java:159)
                at net.bytebuddy.TypeCache$WithInlineExpunction.findOrInsert(TypeCache.java:353)
                at org.mockito.internal.creation.bytebuddy.TypeCachingBytecodeGenerator.mockClass(TypeCachingBytecodeGenerator.java:32)
                at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.createMockType(InlineByteBuddyMockMaker.java:200)
                at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.createMock(InlineByteBuddyMockMaker.java:181)
                at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:35)
                at org.mockito.internal.MockitoCore.mock(MockitoCore.java:65)
                at org.mockito.Mockito.mock(Mockito.java:1875)
                ... 7 more

                Caused by:
                java.lang.IllegalArgumentException: Unsupported class file major version 55
                    at net.bytebuddy.jar.asm.ClassReader.<init>(ClassReader.java:166)
                    at net.bytebuddy.jar.asm.ClassReader.<init>(ClassReader.java:148)
                    at net.bytebuddy.jar.asm.ClassReader.<init>(ClassReader.java:136)
                    at net.bytebuddy.utility.OpenedClassReader.of(OpenedClassReader.java:54)
                    at net.bytebuddy.dynamic.scaffold.TypeWriter$Default$ForInlining.create(TypeWriter.java:2912)
                    at net.bytebuddy.dynamic.scaffold.TypeWriter$Default.make(TypeWriter.java:1634)
                    at net.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:171)
                    at net.bytebuddy.dynamic.scaffold.inline.AbstractInliningDynamicTypeBuilder.make(AbstractInliningDynamicTypeBuilder.java:92)
                    at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase.make(DynamicType.java:2669)
                    at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.transform(InlineBytecodeGenerator.java:183)
                    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
                    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
                    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
                    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
                    at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
                    at org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator.triggerRetransformation(InlineBytecodeGenerator.java:119)
                    ... 20 more

1 test completed, 1 failed

Extremely poor performance compared to docker-compose

I'm not sure what the cause of this is, but if I run phpunit in batect, it takes 3.84 minutes compared to 6.1 seconds in docker-compose. This is not including the build time for the containers, only the time taken by phpunit itself (it outputs it at the end of the tests).

Both of them are not using any volume mounts, and are using the same Dockerfile.

Example of customising shell inside container

What a keen tool, love it, thank you.

Some documentation requests:

• Help setting up the first time shell (command: bash). For example, explaining how to bring in a user's .bashrc, .bash_profile, etc.
• Tips on integrating common shell setups, such as one of the fabulous "git" prompts. https://github.com/magicmonty/bash-git-prompt is a popular example

I'm starting a new project for client, and look forward to suggesting batect to the teams.

False error and no log output if container exits too quickly

I'll prefix this by saying that I haven't been able to replicate it on my MacBook, but it's happening consistently on multiple CentOS machines.

If running a task that exits almost immediately (for example, where the command is echo "hi", instead of container output, I get:

Error: could not run container xxx
Resizing TTY for container 'xxx' failed: bad file descriptor: unknown

with a failure message.

If I make a task to shell into the container and run the same command/script, it passes.

If I make the task run longer (for example, changing my echo.sh to have echo "hi" && sleep 5), I correctly see the log output and batect completes without failure.

This sounds related to #19. It seems to be a regression as this didn't used to occur - whether that's due to a change in batect, Docker, or something else about my environment, I'm not sure.

batect version: 0.25
JVM version: Oracle Corporation OpenJDK 64-Bit Server VM 1.8.0_191
OS version: Linux 3.10.0-862.14.4.el7.x86_64 (amd64)
Docker version: 18.9.2 (API version: 1.39)

Improve error message for Java versions less than 8

When Java version was 1.7 I got the following error:

[Q&A] Kubernetes as backend?

What would it take to take (parts) of Batect to use kubernetes as runtime?

That could be useful if people wanted to escape their constrained workstation resources and run in the cloud.

Something relevant or inspiring might be argo.

Error message when dependency is duplicated

When you create a dependency more than once in batect.yml, you get an error message similar to the following

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Duplicate value 'exchange-rate-service'
 at [Source: (sun.nio.ch.ChannelInputStream); line: 35, column: 9] (through reference chain: batect.config.io.ConfigurationFile["containers"]->java.util.LinkedHashMap["international-transfers-service"]->batect.config.io.ContainerFromFile["dependencies"])

Inputs on CLI get printed twice

Issue discovered when sshed onto a docker container started by batect (bash). Typed commands look like I have very shaky hands/or too much coffee.

e.g. make -> mmaakkee

It's pretty hard to read.

Batect version: 0.23
Container Image: Alpine 3.8

Weird dependency execution order

I've run into a situation where tasks are executing in an unexpected fashion. I've replicated this situation with the batect.yml below.

Expected behaviour: makeTheSandwich runs before takeABiteOfTheSandwich

Is there something I'm missing in my dependency tree that's causing it to run in this order?

batect.yml.txt

One-off tasks against a dependent container

I have a service and a local S3 instance. I need to run an aws-cli command to create a bucket before my app can run. What I would like to do is:

• Launch the local S3
• Run aws-cli command to create a bucket
• Launch my local server

How I’ve attempted it:

containers:
  s3: ...
  aws-cli: ...
  service: ...

tasks:
  start:
    container: service
    dependencies:
      - s3
    prerequisites:
      - createS3Bucket
   
  createS3Bucket:
    container: aws-cli
    run:
      command: aws s3 mb s3://example

This doesn't work because the prerequisites are executed before the task's dependencies are started.

Then I tried adding the S3 container as a dependency to the createS3Bucket task:

containers:
  s3: ...
  aws-cli: ...
  service: ...

tasks:
  start:
    container: service
    dependencies:
      - s3
    prerequisites:
      - createS3Bucket
   
  createS3Bucket:
    container: aws-cli
    dependencies:
      - s3
    run:
      command: aws s3 mb s3://example

This also doesn't work, because when createS3Bucket is complete it tears down the s3 container, and then start spins up a new one.

It'd be great if Batect had a way to model this flow.

Improve the error message when the format of a property is incorrect

When defining a property with the incorrect format, I get the following error message:

/batect-sample-java/batect.yml (line 91, column 18): Cannot deserialize value of type `int` from String "d6001": not a valid Integer value
Caused by: com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `int` from String "d6001": not a valid Integer value
 at [Source: (sun.nio.ch.ChannelInputStream); line: 91, column: 18] (through reference chain: batect.config.io.ConfigurationFile["tasks"]->java.util.LinkedHashMap["app"]->batect.config.io.TaskFromFile["run"]->batect.config.io.TaskRunConfigurationFromFile["ports"]->java.util.HashSet[0]->batect.config.PortMapping["local"])

I'm modified the property ports.local in the batect.yaml file for the batect-sample-java project

...

run:
      container: international-transfers-service
      ports:
        - local: d6001
          container: 6001

....

Invalid project name with build_directory results in obscure Docker error

If you specify a project name that includes upper case letters or spaces, and use a container that uses build_directory, when the container is built you get a relatively unclear error message:

Building image failed: invalid reference format: repository name must be lowercase

It's not immediately obvious that the project name ends up being used as part of the docker image (although in hindsight it makes sense).

Improve the error message when defining empty containers/tasks

When defining a container with an empty property (nodejs), I get the following error message :

my-project/batect.yml: Could not load configuration file: null
Caused by: java.lang.NullPointerException

Here is my batect.yaml:

project_name: my-project

containers:
  nodejs:

tasks:
  start:
    run:
      container: nodejs
      command: ./go.sh

It also happens for empty tasks (start):

project_name: my-project

containers:
  nodejs:
    build_directory: nodejs

tasks:
  start:

Error message given when indentation in configuration file is incorrect is difficult to understand

If you have a configuration file with incorrect indentation, such as:

tasks:
  run:
    description: Run the application
      prerequisites: # This line shouldn't be indented, it should be at the same level as 'description'
      - setup
    run:
      container: build-env
      command: yarn start

then you get an error message like this:

/home/user/Repositories/project/batect.yml (line 27, column 18): mapping values are not allowed here
in 'reader', line 28, column 20:
          prerequisites:
                       ^
 
Caused by: com.fasterxml.jackson.databind.JsonMappingException: mapping values are not allowed here
in 'reader', line 28, column 20:
          prerequisites:
                       ^
 
at [Source: (sun.nio.ch.ChannelInputStream); line: 28, column: 20]
at [Source: (sun.nio.ch.ChannelInputStream); line: 27, column: 18] (through reference chain: batect.config.io.ConfigurationFile["tasks"])

...which is not very clear at all. It would be better if it led the user to what the real issue is.

UnsupportedOperationException when running on Java 9

when trying to run batect -v
the output is:
java.lang.UnsupportedOperationException: Retrieving groups by name is not supported on this platform.

First time user install confusion

The instructions 'Download the latest version of batect from the releases page, and copy it into your project.' are correct, but the releases page contains 4 artefacts where it seems only the script is needed to download. Strictly speaking, this is exactly what the above instruction is telling people, but I considered downloading the JAR as well, which seems redundant.

p.s This is an extremely minor thing.

Batect doesn't work with Java 10

Returns error:

./batect: line 62: ((: 0 2018-04-17: syntax error in expression (error token is "2018-04-17")

Output of java -version:

java version "10.0.1" 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)

Update JSON schema to draft-07

See ineffyble/batectify#2.

Need to make sure this doesn't break any IDEs.

Batect doesn't work on Windows

Hi there!

My environment is:
Java: 1.8.0_191
Docker: 18.09.0
Windows 10
GitBash

When I try to run ./batect --version I have the following error:

Fatal exception:
java.lang.UnsatisfiedLinkError: A opera▒▒o foi conclu▒da com ▒xito.
at jnr.ffi.provider.jffi.NativeLibrary.loadNativeLibraries(NativeLibrary.java:87)
at jnr.ffi.provider.jffi.NativeLibrary.getNativeLibraries(NativeLibrary.java:70)
at jnr.ffi.provider.jffi.NativeLibrary.getSymbolAddress(NativeLibrary.java:49)
at jnr.ffi.provider.jffi.NativeLibrary.findSymbolAddress(NativeLibrary.java:59)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:158)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:89)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:325)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:304)
at batect.os.NativeMethods.(NativeMethods.kt:37)
at batect.InjectionConfigurationKt$osModule$1$1.invoke(InjectionConfiguration.kt:193)
at batect.InjectionConfigurationKt$osModule$1$1.invoke(InjectionConfiguration.kt)
at org.kodein.di.bindings.Singleton$getFactory$1$1$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.SingletonReference.make(references.kt:34)
at org.kodein.di.bindings.Singleton$getFactory$1$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.Singleton$getFactory$1$1.invoke(standardBindings.kt:98)
at org.kodein.di.bindings.StandardScopeRegistry.getOrCreate(scopes.kt:62)
at org.kodein.di.bindings.Singleton$getFactory$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.Singleton$getFactory$1.invoke(standardBindings.kt:98)
at org.kodein.di.KodeinContainer$provider$$inlined$toProvider$1.invoke(curry.kt:14)
at org.kodein.di.internal.DKodeinBaseImpl.Instance(DKodeinImpl.kt:30)
at batect.InjectionConfigurationKt$uiModule$1$4.invoke(InjectionConfiguration.kt:237)
at batect.InjectionConfigurationKt$uiModule$1$4.invoke(InjectionConfiguration.kt)
at batect.InjectionConfigurationKt$uiModule$1$$special$$inlined$singletonWithLogger$1.invoke(KodeinLoggingExtensions.kt:33)
at batect.InjectionConfigurationKt$uiModule$1$$special$$inlined$singletonWithLogger$1.invoke(KodeinLoggingExtensions.kt)
at org.kodein.di.bindings.Singleton$getFactory$1$1$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.SingletonReference.make(references.kt:34)
at org.kodein.di.bindings.Singleton$getFactory$1$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.Singleton$getFactory$1$1.invoke(standardBindings.kt:98)
at org.kodein.di.bindings.StandardScopeRegistry.getOrCreate(scopes.kt:62)
at org.kodein.di.bindings.Singleton$getFactory$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.Singleton$getFactory$1.invoke(standardBindings.kt:98)
at org.kodein.di.KodeinContainer$provider$$inlined$toProvider$1.invoke(curry.kt:14)
at org.kodein.di.internal.DKodeinBaseImpl.Instance(DKodeinImpl.kt:30)
at batect.InjectionConfigurationKt$uiModule$1$3.invoke(InjectionConfiguration.kt:236)
at batect.InjectionConfigurationKt$uiModule$1$3.invoke(InjectionConfiguration.kt)
at org.kodein.di.bindings.Singleton$getFactory$1$1$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.SingletonReference.make(references.kt:34)
at org.kodein.di.bindings.Singleton$getFactory$1$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.Singleton$getFactory$1$1.invoke(standardBindings.kt:98)
at org.kodein.di.bindings.StandardScopeRegistry.getOrCreate(scopes.kt:62)
at org.kodein.di.bindings.Singleton$getFactory$1.invoke(standardBindings.kt:130)
at org.kodein.di.bindings.Singleton$getFactory$1.invoke(standardBindings.kt:98)
at org.kodein.di.KodeinContainer$provider$$inlined$toProvider$1.invoke(curry.kt:14)
at org.kodein.di.internal.DKodeinBaseImpl.Instance(DKodeinImpl.kt:30)
at batect.Application.runCommand(Application.kt:95)
at batect.Application.run(Application.kt:55)
at batect.ApplicationKt.main(Application.kt:36)

I also tried to run batect jar with java -jar batect-0.23.3.jar --version, but I had the same problem.

YAML deserialisation errors are obscure

I accidentally defined volumes as a string instead of an array:

container:
  volumes: .:/code

The resulting error isn't the clearest:

batect.yml: Could not load configuration file: Size must be known in advance when using READ_ALL

I assumed it was a syntax error in my YAML, but from a YAML perspective it's correct. So it must be when it's trying to deserialize valid YAML into an incompatible data structure.

Consider using ".batect" as example folder name instead of "dev-infrastructure"

This is an observation from a couple of projects.

Containing your Dockerfile definitions for batect containers in .batect/ makes them tidier/more out of the way, especially when they're being used purely to run tasks (and there's not any Dockerfile definitions for your actual app, for example).

It also is less confusing than dev-infrastructure which leads my mind to cloud servers and Terraform.

Error message when container entry is duplicated

When you create a second container entry in batect.yml, you get the an error similar to the following

batect.yml (line 46, column 3): Duplicate field 'docker-push-env'
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Duplicate field 'docker-push-env'
 at [Source: (sun.nio.ch.ChannelInputStream); line: 46, column: 18]
 at [Source: (sun.nio.ch.ChannelInputStream); line: 46, column: 3] (through reference chain: batect.config.io.ConfigurationFile["containers"])

Exiting Docker shell failed

Using batect 0.24.0. Upgraded from 0.23.0 by copying a newer shell script version over the older one. This task:

tasks:
    shell:
        run:
            container: build-env
            command: bash --login

Produced this error when exiting:

build-env: running bash --login

boxley@build-env:/src$ exit
logout

Error: An unexpected exception occurred during execution.
java.lang.RuntimeException: Invoking 'stty 6b02:3:4b00:200005cb:4:ff:ff:7f:17:15:12:ff:3:1c:1a:19:11:13:16:f:1:0:14:ff' failed with exit code 1: stty: 'standard input': unable to perform all requested operations

Clean up: done

The task shell failed. See above for details.

Migrate build to Kotlin DSL

I'm uncertain from the roadmap the degree to which this is a Kotlin project. Lately I've been trying Gradle 5.0 with Kotlin DSL 5.0 on small projects. Migrating was straight-forward:

https://guides.gradle.org/migrating-build-logic-from-groovy-to-kotlin/

If you have an interest, I will find some time for a PR to your repo.

Error message when using invalid YAML config file

When I incorrectly indent one line in the batect.yml, I get the following error:

batect.yml (line 20, column 3): while parsing a block mapping
 in 'reader', line 1, column 1:
    project_name: international-tran ... 
    ^
expected <block end>, but found BlockMappingStart
 in 'reader', line 20, column 3:
      database:
      ^

Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while parsing a block mapping
 in 'reader', line 1, column 1:
    project_name: international-tran ... 
    ^
expected <block end>, but found BlockMappingStart
 in 'reader', line 20, column 3:
      database:
      ^

 at [Source: (sun.nio.ch.ChannelInputStream); line: 20, column: 3]

Project level environment variables

Hello,

We can define default variables on container and task with the use of ${MY_VAR:-default_value}. In my case I use the variable at two different places like that:

project_name: test

containers:
  build:
    build_directory: .
    environment:
      DB_USER: ${DB_USER:-admin}
      DB_PASSWORD: ${DB_PASSWORD:-admin}
      DB_DATABASE: ${DB_DATABASE:-test}
  db:
    image: postgres:9.6
    environment:
      POSTGRES_USER: ${DB_USER:-admin}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-admin}
      POSTGRES_DB: ${DB_DATABASE:-test}

Would be easier to set the default value at project level instead of everytime i'm using it. Like that:

project_name: test
environment:
      DB_USER: ${DB_USER:-admin}
      DB_PASSWORD: ${DB_PASSWORD:-admin}
      DB_DATABASE: ${DB_DATABASE:-test}

containers:
  build:
    build_directory: .
    environment:
      DB_USER: ${DB_USER}
      DB_PASSWORD: ${DB_PASSWORD}
      DB_DATABASE: ${DB_DATABASE}
  db:
    image: postgres:9.6
    environment:
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ${DB_DATABASE}

If they are not defined on the host env variable they will get the default value set at the top of the project and be used everywhere else with that value. Do you think that's an interesting feature ? Or may be there is already a way to doing it ?

.dockerignore is incorrectly parsed

Given a .dockerignore file:

# Ignore everything
*

# But what docker actually needs
!.cache
!ci
!package.json
!requirements
!src
!typings
!yarn.lock

and a batect config:

...
    build_directory: .
    dockerfile: ci/server.dev.Dockerfile
    run_as_current_user:
        enabled: true
        home_directory: /home/code
    volumes:
        - container: /code/src
          local: src
          options: cached
    working_directory: /code

The error message:

Error: Could not build image from directory '...'.
Building image failed: Cannot locate specified Dockerfile: ci/server.dev.Dockerfile

Removing the .dockerignore file makes everything work.

I would try to debug the cause and make a PR but don't speak kotlin yet :(

NoSuchElementException: Element Error is missing

I'm receiving the following error when running a task in GoCD:

build succeeds/fails when running multiple times in a row

In project https://github.com/binkley/docker-java11, at commit 4194e19f64c1bd5dde56117badc139c6d52060b5, I can reproduce alternating success/failure with (output elided):

$ ./batect build
$ ./batect build
$ ./batect build
$ ./batect build

Failure looks like:

BUILD SUCCESSFUL in 15s
3 actionable tasks: 3 up-to-date

Error: An unexpected exception occurred during execution.
java.lang.RuntimeException: Invoking 'stty 6b02:3:4b00:200005cb:4:ff:ff:7f:17:15:12:ff:3:1c:1a:19:11:13:16:f:1:0:14:ff' failed with exit code 1: stty: 'standard input': unable to perform all requested operations

Clean up: done

The task build failed. See above for details.

Improve message produced after running upgrade

After running the upgrade task, you get notified that batect has been upgraded which could be read to imply that both the wrapper script and the jar file have been upgraded whereas only the wrapper script has been upgraded to the latest version.

The upgrade message should state explicitly that only the wrapper is being upgraded and that the jar file will be upgraded when you next run batect

Shell tab completion

It would be amazing if batect supported bash/shell autocompletion (e.g. so I can type ./batect bu, hit , and have it autocomplete to ./batect build).

Hot reload / file watch functionality

It would be neat if I could set an option, and have Batect automatically re-build and run the current task on file change.

Ask user to confirm command and run it after making correction suggestion

As a user,
When I mis-type the task name
And...batect gives you a suggestion of what I mostlikely meant,
I want batect to ask me to confirm if that is the command i meant (probably with y/n) and if i confirm...i want batect to run that command

Add support for local proxies

Hi,
I am using cntlm to go through a proxy.
The environment variables that I have set in my host are something like this:

https_proxy=http://localhost:3333
http_proxy=http://localhost:3333
...

I have to set the variables in batect.yml to something like this:

....
   environment:
      - https_proxy=http://host.docker.internal:3333
      - http_proxy=http://host.docker.internal:3333
..

Otherwise batect.yml picks up the ones set in the environment which wouldn't work within the container.

Support Google Cloud Container Registry credential helper

Pulling images from the Google Cloud Container registry fails with the following error:

Could not build image: The credentials returned for 'eu.gcr.io' by the credential helper executable 'docker-credential-gcloud' are invalid: there is no 'ServerURL' field.

The reason seems to be because the Username reported by the credential helper is _dcgcloud_token instead of the expected <token>.

This is the extract from the batect log file (replaced the actual secret with "xxx"):

{
    "@timestamp": "2019-05-22T14:50:35.754Z",
    "@message": "Process exited.",
    "@severity": "debug",
    "@source": "batect.os.ProcessRunner",
    "@processId": 14167,
    "@threadId": 14,
    "@threadName": "pool-1-thread-1",
    "output": "{\n  \"Secret\": \"xxx\",\n  \"Username\": \"_dcgcloud_token\"\n}\n",
    "exitCode": 0,
    "command": [
        "docker-credential-gcloud",
        "get"
    ]
}

Tasks that run on the host

It would be great if you could define a task that executes on the host (not in a Docker container).

This is useful because:

• It means you can use batect as your task runner for all tasks, even the ones that can't/shouldn't run in Docker
• It means when you're setting up task workflows using dependencies, you can include these

A prime example: I want to run a bash script that build a Docker image for my Java app. The Dockerfile includes a COPY that takes my app.zip, which was built by Gradle, and extracts it into the image.

I want to use that bash script/task as a dependency of a task that then uses the new Docker image as part of a Terraform deployment.

This means my task flow goes:

1. Compile the Java (done in a batect java-env)
2. Run shell script to build and push the Docker image
3. Run the Terraform (done in a terraform-env)
4. More steps

Right now, 2. is done in a "docker-env" which actually doesn't contain much, and mounts the Docker socket from the host. Ideally it would be run directly on the host, but I can't do that and still have my end-to-end task workflow with batect.

Add support for --init to make using Node easier

I am running a Node express server with batect. When I input ctrl + c in the terminal, it does not kill the process, instead it just hangs.

With the attached project by running ./batect api and then ctrl +c I'm able to reproduce the bug.
sigterm-error-project.zip

I've also experienced this error with an nginx container but instead of hanging it displays the following message, but nothing happens after that:

Error: Interrupt received during execution.
User interrupted execution.

Behaviour of command without shell entrypoint is unexpected

I've tried several YAML variations on quoting multi-line strings. Latest is:

    run:
      container: build-env
      command: './gradlew assemble
            && java -jar build/libs/docker-java11-0.0.1-SNAPSHOT.jar'

In each case, batect complains:

Task '&&' not found in root project 'docker-java11'.

I'm sure this is PEBKAC. What am I doing wrong?

More logging options

It can be very difficult to troubleshoot things when you're using batect.

As an example, if I have a task that start my app code, and has a dependency on a localstack container, I only have access to the app logs (via stdout). I have no way to see the logs of the localstack container at all.

Things that would be useful:

• An option to keep stopped containers so I can docker logs them, even if they don't fail
• The ability to interleave logs from all running containers to stdout (like docker-compose does)
• An option to save container logs to a file

Add support for environment variables in build_args

As https://docs.docker.com/compose/compose-file/#args has the feature

You can omit the value when specifying a build argument, in which case its value at build time is the value in the environment where Compose is running.

It would be really useful here as well. Our use case is we pass a GITHUB_TOKEN in order to install private dependencies, to a multi-stage Dockerfile.

See #68 for the same feature but for environment.

Nicer formatting for --list-tasks

Our project has 63 tasks defined in our batect.yml, and as a result the output of batect --list-tasks can be a bit of a mess.

Something that our team found handy was adding "\n\t" to the start of every description - this means the description for each task is on a newline and indented, which greatly improves visibility.

Perhaps something like this could be incorporated into batect natively?

Published ports do not work if port is not exposed in Dockerfile

If a container or task publishes a port to the local machine, but this port does not have a corresponding EXPOSE instruction in the image's Dockerfile, the port is not published and is not accessible on the host machine.

This works correctly in v0.17, but not v0.18.

Support for YAML aliases

Something I've found very helpful with writing build pipelines for GoCD is using YAML aliases - which let you define a block of code (an object or array) once, and then reuse it in multiple places, without rewriting.

This would be very useful for helping to make our batect.yml smaller and more readable (it's currently over 600 lines!).

I tried to use a YAML alias, but got a message that YAML aliases are not supported by whichever YAML parser you're currently using.

Support build context

I can't seem to be able to replicate the structure that we currently have. We have docker/php/Dockerfile, but I need the image to have access to . as the build context.

Am I missing something here?

Received error message when trying to run a particular batect task after upgrading

I'm getting the following error when trying to run a particular batect task:

Test Suites: 15 passed, 15 total
Tests:       6 skipped, 91 passed, 97 total
Snapshots:   0 total
Time:        9.182s
Ran all test suites.

Clean up: done

Running unit-test...
build-env: running true

Error: Could not run container build-env.
Resizing TTY for container '0d60dfbf9a42a41342b51152a8376f67f1f8c20aa834fc561a436de4fd3daadd' failed: cannot resize a stopped container: unknown

Clean up: done

The task unit-test failed. See above for details.

The above unit test ran but I got an error at the end

I'm finding that tasks which do not include a prerequisite property will work fine but ones that do, will not. For example this works:

unit-test-ui:
    description: Run unit tests for UI
    run:
      container: build-env
      command: scripts/unit-test.sh

But this does not:

unit-test:
    description: Run all unit tests for both UI and BFF
    prerequisites:
      - unit-test-backend
      - unit-test-ui
    run:
      container: build-env
      command: "true"

Improve error message when defining a container with an unknown field

When defining a container with an unknown property (non-existant), I get a confusing error message:

my-project/batect.yml (line 6, column 3): Unknown field 'non_existant'
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "non_existant" (class batect.config.io.ContainerFromFile), not marked as ignorable (10 known properties: "ports", "environment", "health_check", "volumes", "run_as_current_user", "image", "build_directory", "working_directory", "dependencies", "command"])
 at [Source: (sun.nio.ch.ChannelInputStream); line: 6, column: 3] (through reference chain: batect.config.io.ConfigurationFile["containers"]->java.util.LinkedHashMap["nodejs"]->batect.config.io.ContainerFromFile["non_existant"])

Here is my batect.yaml:

project_name: my-project

containers:
  nodejs:
    non_existant: nodejs

tasks:
  start:
    run:
      container: nodejs
      command: ./go.sh

Maybe interesting: Toast, written in Rust

Hi @charleskorn ,

I had a look at this tool, Toast, and it looked somewhat similar to Batect. One major difference seems to be that there is only a single (shared) image across tasks, and that there is no orchestration across containers, limiting its use for tests.

The reason I post it here is to possibly give you inspiration for further developing Batect.

Cheers,
Sebastian

Allow adding or dropping Docker capabilities

The container spec should allow specifying Docker capabilities. See docker-compose documentation for an example implementation.

This functionality would be useful:

A) When wanting to tighten security, by allowing the user to drop unused capabilities

B) When needing to run a Docker container/task with extra capabilities (for example, adding SYS_MODULE to enable something in a container to load a kernel module)

I would suggest the privileged flag should also be part of this flag. This enables a range of extra privileges on a container, and is used for things like accessing host devices.

Moving to Kotlin/native, away from JVM: stories

I'm excited to see in your "Roadmap" doc (first rate docs, btw) that you intent to move to Kotlin/native. This appears a sound choice.

I looked through the issue tracker, and didn't see much around this aspiration. If you could post particular areas of batect where you'd like to see the migration begin, it might garner volunteers.

More concise passing of environment variables

Our batect.yml currently has many many lines under environment blocks that look like:

environment:
    HOSTNAME: $HOSTNAME

When you want to pass a host variable through to the container as the same name and value, a more concise format could be helpful. This is something Docker-Compose and Buildkite do well.

E.g.:

environment:
    - HOSTNAME