landregistry/common-dev-env

A generic Docker/Docker Compose-based development environment with some useful hooks to get apps up and running faster

common-dev-env's Issues

Unable to find Compose fragments on Windows

COMPOSE_FILE needs semicolons in windows:

https://docs.docker.com/compose/reference/envvars/#compose_file

Files created in a container are owned by root user on host filesystem

• I'm submitting a ...

  • bug report
  • feature request
  • support request

(Debatable!?)

• What is the current behavior?
  Files written inside the container arrive on the host file system owned by the root user.

• If the current behavior is a bug, what are the steps to reproduce?
  bashin to a container. Use touch to create a new file. Navigate to directory on host filesystem and observe that files are owned by root.

• What is the expected/proposed behavior?
  Files should be written as my current user, not root.

• What is the motivation / use case for changing the behavior?
  If you want to remove files that were created in the container you have to sudo rm them. Causes problems particularly with the flask-skeleton-ui which runs the build script in the container.

• Please tell us about your environment:

  • Common-dev-env version: 1.0.5
  • Operating system (and shell, if not the OS default): Ubuntu 18.10 + Zsh
  • Ruby version: 2.5.1p57

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  Run docker with the -u flag to specify the current user instead of root

Update versions of builtin commodities

Postgres, nginx etc

Support for running linting and formatting scripts consistently

• I'm submitting a ...

  • feature request

• What is the current behavior?
  Currently, the onus of running linting and formatting commands is upon the individual developers themselves. They are required to select out the appropriate commands and run them in the correct order, so that they comply with the organisations code style requirements

• What is the expected/proposed behavior?
  Introducing two new aliases; lint and format. These should execute their respective makefiles, in a similar way to the integrationtest and unit-test aliases. This is to allow for applications agnosticism (as different languages will need different tools)

• What is the motivation / use case for changing the behavior?
  -- Making it easier for developers to lint and format their code by making it a single command.
  -- Allow for teams to enforce consistent linting. It is really easy for developers to accidentally run the required commands in the wrong order, or to accidentally use the wrong settings, etc. By using an alias, teams will have to worry less about inconsistent linting and formatting, so long as developers use the appropriate alias.
  -- Land Registry should be encouraging common-dev-env users to lint and format their code as part of good coding practice, and doing this gives a clear steer that this is required functionality for every project and team

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  The scripts themselves, in general, should not be too complex - they are after all just executing make commands. I do however suggest that, at least for the lint command, the following flags are added:

  -- An -r flag for generating reports, in a similar vein to how unit-test generates reports. Most linters output long reams of reporting, and it could be useful for that to be put into a file (particularly if it needs to be shared)
  -- An -f flag for automatic fixes. A useful feature implemented in some linters such as ESLint is to automatically fix issues where possible. These auto fixes are also typically configurable within-command however, and so having a dedicated flag for this is important to retain consistent usage. For other linters that do not support auto fixing, developers of applications could use the flag to run other tools that provide some fixes before executing the main linter (such as executing black commands before running flake8). The implementation of the alias should not presume however that the format command should be ran before executing the linter, as that may not always be appropriate; the flag should be passed directly into the make command for developers to handle how they want.

  The crux of this feature suggestion is to make linting and formatting both more available and more consistent. As a result, I also suggest that arguments are rejected unless they happen to be part of a flag that we handle ourselves (as outlined above). If we start allowing developers to put additional arguments through the aliases, then we'll find divergence in how linting/formatting between developers in the same team again, rendering the whole exercise pointless. Using different arguments should be an active decision, whether through making changes to the application's Makefile, or through directly using the linting/formatting commands (which should be discouraged where possible)

Postgres initial setup fails due to fast loops

Intended behaviour

Once the postgres commodity is found to be required by one of the apps its image is downloaded and ran. While the container is starting the wait-for-it.sh script is ran. The script checks periodically if a psql command can be ran. This results in the script waiting for the container to be up before running the initialization script(postgres-init-fragment.sql)

The problem

The problem occurs due to the small time in between periodic checks (as of currently 1 second) a psql command may be able to return while the database is not yet fully responsive. This then causes the initialization script to be ran before the database is able to receive and process the script as intended. Due to the nature of how the .commodities.yml file works if this problem occurs there is no easy way in order to reinitialise the database as the script itself will avoid performing the setup as it assumes the commodity was already setup.

Evidence

As evidence I can provide the relevant logs of the source run.sh up command:

Successfully tagged dv_postgres:latest
Creating postgres ... done

Waiting for postgres to finish initialising
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Postgres is unavailable - sleeping
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Postgres is unavailable - sleeping
Postgres is ready
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Searching for alembic code

As you can see as expected the first response is a could not connect to server as expected as the until loop runs straight away. However due to short sleep time in between attempts we get the Postgres is ready message, this should indicate that the script has been completed and therefore the initialization script can be ran, but as it can be seen on the logs postgres fails another two attempts of the psql command before the script follows on to search for the following commodities.

Replication of issue

The problem is not guaranteed to happen every single time. This is due to the fact that postgres may boot time is not constant and its boot may align in such a way with the loop that there is a full second of difference between two runs of the script. This results in the database sometimes being able to receive the query and other times not.

Solution

The solution that I am proposing is not perfect but makes the system much more reliable with minimal changes. With testing I was capable of solving this problem by increasing the time in between attempts to 5 seconds. This resulted in the intended behaviour with the database being properly setup and the initialization script running without problems.

The obvious drawback of such solution is that the setup time is now increased. Nevertheless, I believe that this is not of major concern due to commodities only needing to be setup once. This will only increase up to 4s(in a worst case scenario) in the initial setup of the environment.

A better solution is to use a more powerful version of wait-for-it.sh such as the version by @vishnubob which can be found here. Although I find such solution overcomplicated for the problem

Put general tooling help (Shells, SSH etc) in wiki

Keeps the README free for information specific to the dev-env

Unable to connect to DB2 database following fresh provision

After provisioning the development environment, I'm unable to connect to the default LANDREG DB2 database.

I've tested this using a full Digital Mortgage development environment and a newly-cloned development environment only pointing at one application. In both scenarios I receive the same response from DB2 when attempting to connect to the default database.

The following is the output from the dev-env when creating the database and running SQL found in the relevant fragment file(s):

Successfully tagged dv_db2:latest
Creating db2 ... done

Found some in migrate-metadata
Waiting for DB2 to finish initialising
DB2 is not up yet
DB2 is up and running
DB2 is available after 1 seconds
Setting up default LANDREG database
CREATE DATABASE LANDREG
DB20000I  The CREATE DATABASE command completed successfully.

DB20000I  The SQL DISCONNECT command completed successfully.
Database setup complete. Looking for table data now.
CREATE DATABASE landreg
SQL1005N  The database alias "landreg" already exists in either the local
database directory or system database directory.

CONNECT TO landreg

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.5.5
 SQL authorization ID   = DB2INST1
 Local database alias   = LANDREG

/// Table creation and inserts from fragment file here report as being run successfully ///

After running the $ db2 command, the following is output:

(c) Copyright IBM Corporation 1993,2007
Command Line Processor for DB2 Client 10.5.5

You can issue database manager commands and SQL statements from the command
prompt. For example:
    db2 => connect to sample
    db2 => bind sample.bnd

For general help, type: ?.
For command help, type: ? command, where command can be
the first few keywords of a database manager command. For example:
 ? CATALOG DATABASE for help on the CATALOG DATABASE command
 ? CATALOG          for help on all of the CATALOG commands.

To exit db2 interactive mode, type QUIT at the command prompt. Outside
interactive mode, all commands must be prefixed with 'db2'.
To list the current command option settings, type LIST COMMAND OPTIONS.

For more detailed help, refer to the Online Reference Manual.

Running the command:

db2 => CONNECT TO LANDREG

Gives the following output:

SQL1031N  The database directory cannot be found on the indicated file system.
SQLSTATE=58031

Ready for use message appears before custom script searches

So it's not actually ready yet...

Add Vagrant support

• I'm submitting a ...

  • feature request

• What is the current behavior?
  You can only run the dev-env on a host with all dependencies installed and full non-sudo control over docker

• What is the expected/proposed behavior?
  Add vagrant support so a machine can be spun up with all dependencies preinstalled at a tested stable level

• What is the motivation / use case for changing the behavior?
  Allows more flexibility in where the dev-env can be used

Support Compose v3 files

• I'm submitting a ... feature request

• What is the current behavior?
  Because all fragments are combined into one via the COMPOSE_FILE env var, they must all be the same Compose file version. And as the commodities use v2, so must the apps, if any use at least one commodity.

• What is the proposed behavior?
  Support v3 compose files, perhaps by allowing alternate file names or detection that all the apps use 3 and changing commodities accordingly. Regardless, might need quite a bit of extra validation code in case of mismatches.

• What is the motivation / use case for changing the behavior?
  While v2 is still supported, v3 gives more options, possible compatibility with docker stack if we go that route, and can simply leverage people's probable knowledge a bit more.

Aliases that rely on using `ex` commands do not work on Windows / Git Bash

• I'm submitting a ...

  • bug report
  • feature request
  • support request

• What is the current behavior?
  Currently, it is impossible to run aliases in scripts/run-aliases.sh that contain ex <container> commands in Windows. This is both due to git bash on windows needing winpty to run docker exec -it commands, but also because executing ex on winpty raises a vi terminal instead of the intended execution in a docker container.

• If the current behavior is a bug, what are the steps to reproduce?

  • Be on Windows OS / Git Bash
  • Up the environment
  • Execute an alias that uses ex in its function (e.g. integration-test <container>)

• What is the expected/proposed behavior?
  The alias is executed successfully

• What is the motivation / use case for changing the behavior?
  Windows developers will not be able to use most aliases otherwise

• Please tell us about your environment:

  • Common-dev-env version: 1.20.0
  • Operating system (and shell, if not the OS default): Windows / Git Bash
  • Ruby version: v2.5+

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  In add-aliases.sh, functions that use ex should have a Windows OS check. If the OS is Windows, then use winpty docker exec -it instead of ex. Otherwise use ex.

This would make a good first issue for someone to pick up, if they have access to a windows machine.

Allow defining tag versions in environment configuration yml

• I'm submitting a ...

  • bug report
  • feature request
  • support request

• What is the current behavior?
  In the configuration.yml file, each application requires a repo and branch to pull down/update.

• If the current behavior is a bug, what are the steps to reproduce?
  n/a

• What is the expected/proposed behavior?
  I'd like to be able to specify a tag instead of a branch, per application, such that sets of versioned applications that work together can be defined in the environment config.

• What is the motivation / use case for changing the behavior?
  Pulling from master or equivalent on each repo may result in unreleased changes that are not compatible with other applications interfaces. In order to avoid having to also state which tag the user/developer should checkout, I'd rather specifiy this in the config.

• Please tell us about your environment:

  • Common-dev-env version: 1.0.6
  • Operating system (and shell, if not the OS default): MacOS Mojave, Zsh
  • Ruby version: 2.4.2

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  n/a

Add the ability to maintain additional compose-fragment.yml files for specific purposes

• I'm submitting a ...

  • bug report
  • feature request
  • support request

• What is the current behavior?
  The common dev-env is only able to pull in a single compose-fragment.yml for each specific respository in the project's dev-env config, resulting in the compose-fragment.yml having to define the entire deployment structure of the repo's contents.

• What is the expected/proposed behavior?
  It is proposed that the ability to maintain additional compose-fragment.yml files for specific purposes be added to the common dev-env. This would be in the following structure:

fragments/compose-fragment.yml # default - would not change
fragments/compose-fragment.slim.yml
fragments/compose-fragment.full.yml

This would then be referenced in the dev-env config configuration.yml in the following manner:

applications:
  # default - would not change
  example-repo:
    repo: [email protected]:example/example.git
    branch: master
  # slim
  example-repo:
    repo: [email protected]:example/example.git
    branch: master
    variant: slim
  # full
  example-repo:
    repo: [email protected]:example/example.git
    branch: master
    variant: full

• What is the motivation / use case for changing the behavior?
  The current behaviour is fine for individual microservice repos, however for monolithic repos containing multiple microservices it is not possible to define individual configs for specific purposes (i.e., if only a subset of components in the repo are required). This can be worked around by having separate branches with redefined compose-fragment.yml files, however this creates additional complexity in terms of repo management.

• Please tell us about your environment:

  • Common-dev-env version: 2.2.0
  • Operating system (and shell, if not the OS default): Fedora (zsh) via WSL2
  • Ruby version: 3.2.2

Vagrant does not support empty compose fragments

• I'm submitting a ...

  • bug report

• What is the current behavior?
  If an app contains an empty compose fragment file, Vagrant quits with a ruby error:

Vagrant failed to initialize at a very early stage:There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
an invalid or undefined variable.Path: D
Line number: 0
Message: undefined method `each'

• If the current behavior is a bug, what are the steps to reproduce?
  See above

• What is the expected/proposed behavior?
  Vagrant should start up normally

• Please tell us about your environment:

  • Common-dev-env version: 1.6.1
  • Operating system (and shell, if not the OS default): Any
  • Ruby version: Any

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  expose_ports.rb, check for services key existence before looping through it

Pluggable Commodities

• I'm submitting a ...

  • feature request

• What is the current behavior?
  Commodities are fixed and use custom code built into the dev env to provision

• What is the expected/proposed behavior?
  Have some sort of plugin based architecture for commodoties, including private sources. Use some sort of API to adhere to to hook in to various parts of the dev env.

• What is the motivation / use case for changing the behavior?
  More flexibility

Shell scripts are not executable by default

Support expensive services that don't persist

• I'm submitting a ...

  • feature request

• What is the current behavior?
  If you have an expensive service that should have staggered startup, but also is a one off job that exits cleanly, there is no healthcheck to exec successfully so the dev env polls forever

• What is the expected/proposed behavior?
  Support some way for the expensive service configuration to allow for exit checks (and exit code?)

• What is the motivation / use case for changing the behavior?
  Support more varied apps

Reduce log spam if no configuration.yml

• I'm submitting a ...

  • feature request

• What is the current behavior?
  If an app doesn't have a configuration.yml this is spammed in the logs alot

• If the current behavior is a bug, what are the steps to reproduce?
  Add an app without a configuration.yml

• What is the expected/proposed behavior?
  One log entry at the start

• What is the motivation / use case for changing the behavior?
  Cleaner startup trace

Windows 10 compatibility

• I'm submitting a ...

  • bug report

• What is the current behaviour?
  There is no combination of tools that allow the dev-env to fully work on Windows 10 if an app is using a commodity or relative paths in the compose fragment as per the example snippet. Git Bash and WSL seem to pass through invalid paths into the Windows Docker engine via the docker-compose-fragment. Also, the COMPOSE_FILE environment variable is set in WSL but not seen by Docker.

• If the current behaviour is a bug, what are the steps to reproduce?
  source run.sh up when using an app that implements the features in the example docker-compose-fragment, using Git Bash or WSL (connecting to Windows Docker)

• What is the expected behaviour?
  It should just work

• Please tell us about your environment:

  • Common-dev-env version: 1.0.4
  • Operating system (and shell, if not the OS default): Windows 10, WSL or Git Bash
  • Ruby version: 2.5.1

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  WSLENV could be the key for both passing through COMPOSE_FILE and translating the paths within it. Whether that conversion would then be used when resolving . in the fragments themselves is unknown. If the whole env var cannot be parsed by WSLENV, perhaps the first dummy compose fragment is all that needs to change as that is used for the . definition

psql alias does not work for postgres-96

• I'm submitting a ...

  • bug report
  • feature request
  • support request

• What is the current behavior?
  The psql alias maps to docker-compose exec postgres psql -h postgres -U root -d which does not work for the postgres-96 service/commodity.

• If the current behavior is a bug, what are the steps to reproduce?
  Use the psql command in a workspace where the postgres-9.6 commodity is used instead of postgres.

• What is the expected/proposed behavior?
  For either psql to work with both commodities/services, or a separate alias to be added, e.g. psql96.

• What is the motivation / use case for changing the behavior?
  A longer, non aliased, docker-compose command has to be used instead. An alias, like the existing psql, would be nice.

• Please tell us about your environment:

  • Common-dev-env version: v1.2.1

Add pull request template

Reminding to set correct destination etc

Post-update message is incorrect

After the update, it says Please rerun your command (source run.sh check-for-update)

This should be Please rerun your command (source run.sh up)

Randomise the Compose project name

Currently the Compose project name is hardcoded to dv, which is used as a prefix for the network and containers. If this was randomised, then multiple dev envs could run simultaneously, (host) port conflicts notwithstanding. Consideration would need to be given to whether the network should continue to be shared, and what should happen with commodities, where port conflict would definitely be a problem.

Allow an alternate branch/ref to be specified for the config repo

• I'm submitting a ...

  • bug report
  • feature request
  • support request

• What is the current behavior?
  When you specify the location of a config repo on first up, the default branch is always used

• What is the expected/proposed behavior?
  Allow an alternate branch/ref to be specified for the config repo itself, similar to app repo refs within the config itself

• What is the motivation / use case for changing the behavior?
  It allows a single config repo to be used for several scenarios

Support for Postgres 17

• I'm submitting a ...

  • feature request

• What is the expected/proposed behavior?
  Support for Postgres 17

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  Due for release late 2024

Swap expensive/inexpensive startup order

• I'm submitting a ...

  • feature request

• What is the current behavior?
  Expensive services start first

• What is the expected/proposed behavior?
  Expensive services start second

• What is the motivation / use case for changing the behavior?
  Allows you to start using the other apps while waiting

Non-docker apps

• I'm submitting a ...

  • feature request

• What is the current behavior?
  If an "app" is cloned down that does not contain a docker compose file, an error occurs, but this may actually be a legit situation e.g. repo file useful files in it

• What is the expected/proposed behavior?
  Allow the dev env config to specify that a given repo does not container a dockerable app

• What is the motivation / use case for changing the behavior?
  More flexibility to provide every resource a developer needs

aliases for tests

• I'm submitting a ...

  • bug report
  • feature request
  • support request

• What is the current behavior?
  Aliases for acceptance tests only allow you to run the cucumber tests and run linting with no parameters.

• What is the expected/proposed behavior?
  Adding a bundlelock alias would allow the gemfile.lock to be updated within the container including passing in parameters to update individual gems.
  The acclint alias could also pass through any parameters to allow auto-fixing of linting errors.

• What is the motivation / use case for changing the behavior?
  Simplicity and consistency.

Local copy of tini

• I'm submitting a ...

  • bug report
  • feature request
  • support request

• What is the current behavior?
  DB2 and logger Dockerfiles download tini from github

• What is the expected/proposed behavior?
  Have a local copy of tini in the repo so as not to rely on third party availability/code

• What is the motivation / use case for changing the behavior?
  Security, reliabiliy

Support compose v2

docker compose instead of docker-compose

https://docs.docker.com/compose/cli-command-compatibility/

No --services --filter

No --compatibility

Possibly more issues

Halt in a new console window produces alias errors

Tries to remove aliases that don't exist. Add them first maybe?

Build failure when handling MkDocs skeleton repositories

• I'm submitting a ...

  • feature request (possible bug report, depending on whether this is something that was previously considered)

• What is the current behavior?
  The development environment fails to build when searching for configuration.yml for repositories based on the mkdocs skeleton.

No configuration.yml found for documentation
Searching for NGINX conf files in the apps
No configuration.yml found for documentation
Searching for Elasticsearch initialisation scripts in the apps
No configuration.yml found for documentation
Searching for elasticsearch5 initialisation scripts in the apps
No configuration.yml found for documentation
logic.rb:261:in `<main>': undefined method `positive?' for 0:Fixnum (NoMethodError)

• If the current behavior is a bug, what are the steps to reproduce?
  Include a repository based on the mkdocs skeleton and attempt to run a fresh source run.sh up.

• What is the expected/proposed behavior?
  To be discussed - should teams include these as part of their dev-env-config, or should we be running these separately? If including them, should the development environment handle running the skeleton, or should it skip over the repository and leave it to devs to run locally?

• What is the motivation / use case for changing the behavior?
  Including local copies of documentation repositories which use the mkdocs skeleton.

• Please tell us about your environment:

  • Common-dev-env version: v1.1.0
  • Operating system (and shell, if not the OS default): macOS
  • Ruby version: 2.2.3p173

Allow dev env config to specify a message after successful startup

• I'm submitting a ...

  • feature request

• What is the expected/proposed behavior?
  Allow dev env config to specify a message after successful startup

• What is the motivation / use case for changing the behavior?
  More info to devs starting up an unfamiliar environment

Cannot run Compose commands from anywhere other than devenv root

• I'm submitting a ...

  • bug report

• What is the current behaviour?
  Due to the use of PWD in docker-compose-fragment files, docker-compose commands fail when run anywhere other than the root of the dev-env, as the variable is re-interpreted every time.

• If the current behaviour is a bug, what are the steps to reproduce?
  cd apps && status

• What is the expected/proposed behaviour?
  It works

• What is the motivation / use case for changing the behavior?
  Working in subdirectories e.g. the apps is a normal part of development

• Please tell us about your environment:

  • Common-dev-env version: 1.0.3
  • Operating system (and shell, if not the OS default): All
  • Ruby version: 2.5.1

• Any other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, etc):
  Remove need for PWD by forcing docker compose context root

Wiremock does not work

/vagrant is in the docker-compose-fragment

Broken link in README.md

The spatial Postgres example link in the README.md is broken - it's pointing at a .sql extension, but the extension of the actual file is .yml. I've fixed this on a local branch, can push it up with repo access.