itv/servicebox

Scala library to control docker containers programmatically

Service box

A library to define and run test dependencies using scala and Docker containers.

Containers and integration testing

Scala's powerful type system can help avoiding a range of otherwise common bugs, removing the need for pedantic, low-level unit testing. However, testing how the various components integrate into a larger system is still necessary. It is in fact at this higher level that errors are typically discovered (e.g. serialisation/deserialisation, missing configuration values, SQL queries working differently across different database vendors, etc.).

Servicebox allows to define external dependencies in idiomatic scala, managing their lifecycle and execution within Docker containers. The main goal is to support developers in writing effective and reliable integration tests, by making easier to setup a CI/CD environment that closely resembles the production one.

Status

This library is at an early development stage and its API is likely to change significantly over the upcoming releases.

Getting started

You can install servicebox by adding the following dependencies to your build.sbt file:

val serviceboxVersion = "<CurrentVersion>"
libraryDependencies ++= Seq(
  "com.itv" %% "servicebox-core" % serviceboxVersion,
  "com.itv" %% "servicebox-docker" % serviceboxVersion, //docker support
)

To start with, you must specify your service dependencies as follows:

import cats.effect.{ContextShift, IO}
import scala.concurrent.duration._
import cats.data.NonEmptyList

import com.itv.servicebox.algebra._
import com.itv.servicebox.interpreter._
import com.itv.servicebox.docker

import doobie._
import doobie.implicits._

import scala.concurrent.ExecutionContext

object Postgres {
  case class DbConfig(host: String, dbName: String, password: String, port: Int)

  implicit val cs: ContextShift[IO] = IO.contextShift(ExecutionContext.global)
  
  val xa = Transactor.fromDriverManager[IO](
    "org.postgresql.Driver", 
    "jdbc:postgresql:world",
    "postgres",
    "" 
  )
  
  def pingDb(value: DbConfig): IO[Unit] = IO {
    sql"select 1".query[Unit].unique.transact(xa)
  } 

  def apply(config: DbConfig): Service.Spec[IO] = {
    // this will be re-attempted if an error is raised when running the query
    def dbConnect(endpoints: Endpoints): IO[Unit] =
      for {
        _ <- IOLogger.info("Attempting to connect to DB ...")
        ep = endpoints.toNel.head
        serviceConfig = config.copy(host = ep.host, port = ep.port)
        _ <- pingDb(serviceConfig)
        _ <- IOLogger.info("... connected")
      } yield ()

    Service.Spec[IO](
      "Postgres",
      NonEmptyList.of(
        Container.Spec("postgres:9.5.4",
                       Map("POSTGRES_DB" -> config.dbName, "POSTGRES_PASSWORD" -> config.password),
                       Set(PortSpec.autoAssign(5432)),
                       None,
                       None)),
      Service.ReadyCheck[IO](dbConnect, 50.millis, 5.seconds)
    )
  }
}

A Service.Spec[F[_]] consists of one or more container descriptions, together with a ReadyCheck: an effectfull function which will be called repeatedly (i.e. every 50 millis) until it either returns successfully or it times out.

Once defined, one or several service specs might be executed through a Runner:

import scala.concurrent.ExecutionContext.Implicits.global

implicit val tag: AppTag = AppTag("com.example", "some-app")

val config = Postgres.DbConfig("localhost", "user", "pass", 5432)
val postgresSpec = Postgres(config)

//evaluate only once to prevent shutdown hook to be fired multiple times
lazy val runner = {
  val instance = docker.runner()(postgresSpec)
  sys.addShutdownHook {
    instance.tearDown.unsafeRunSync()
  }
  instance
}

The service Runner exposes two main methods: a tearDown, which will kill all the containers defined in the spec, and a setUp:

val registeredServices = runner.setUp.unsafeRunSync()

This returns us a wrapper of a Map[Service.Ref, Service.Registered[F]] providing us with some convenience methods to resolve running services/containers:

val pgLocation = registeredServices.locationFor(postgresSpec.ref, 5432).unsafeRunSync()

Notice that, while in the Postgres spec we define a container port, the library will automatically bind it to an available host port (see InMemoryServiceRegistry for details). Remember that, in order to use the service in your tests, you will have to point your app to the dynamically assigned host/port

pgLocation.port

Detailed example

Please refer to this module for a more detailed usage example illustrating how to integrate the library with scalatest.

Key components

The library currently consists of the following modules:

• An "algebra" to define test dependencies (aka Service) as aggregates of one or several Container.
• An InMemoryServiceRegistry that can automatically assign available host ports to a service containers.
• A Scheduler, which provides a simple interface suitable to repeatedly check if a service is ready
• "Interpreters" to setup/check/teardown services using Docker as container technology, and scala.concurrent.Future or cats.effect.IO as the effect system.

Modules

• core: the core algebra, with built-in support for scala.concurrent.Future.
• core-io: optional support for cats.effect.IO
• docker: a docker interpreter for the core algebra.

servicebox's Issues

Adding scala test integration for servicebox

In order to facilitate testing and make service box easier to use in such context, one could provide a set of methods in order to reduce boilerplate.

I propose something like the following methods:

def withServicesUp[F[_]](specs: Spec[F] *)(f: Map[(Service.Ref, Port), Location] => F) : Unit
def withServicesUpUnsafe[F, R](specs: Spec[F] *)(f: Map[(Service.Ref, Port), Location] => R)  : Unit

These methods would essentially

1. start the application
2. wait for all services to be healthy
3. Attempt to run the test defined by f
4. Cleanup the docker state (stopping && removing containers )
5. Propagate the effect final result back to the testing framework

This would potentially allow users to create tests as following

def withApp = ??? //some code that start my application under test
"My app" when "ready" should withServiceBoxUp { config =>
  val dbPort = config(Postgres.ref, Port(5432)).port
  "obey a rule" withApp(config) { app =>
       //testCode goes here
  }

  "obey another rule" withApp(config) { app =>
       //testCode goes here
  }
}

Because we don't want these to clutter the current modules of service box I would also propose adding them in a separate module.