This repository aims to streamline how every docker based Indico project is buildt. The repository comprises a script (build.sh)[./build.sh] which kan be added to all Indico projects, as well as clear guides and best practices for structuring, versioning, and working on Indico projects using Git.
As a general rule every project should try to follow the standard versioning approach for the language chosen for the project.
How a project is versioned is usually determined by the package-manger or framework used by the project. As we try to
mitigate the need for custom logic in each project, we try to follow the best practises for versioning the individual
projects determined by the package-manager or framework. As of 22.02.22
Indico has 3 projects that targets a container-based
environment:
- FT4: Ferdig til 4, is the main backend API written in PHP and JavaScript.
- WM4: Worker Manager for FT4, is the file-processing and queue handler for FT4 written in PHP and Python.
- Vault: Evidence Vault, is the frontend for FT4 written in TypeScript with Svelte.
For Vault this is pretty straight forward as it uses NPM as its package-manager which also
handles versioning through its package.json
. The package.json
also contains relevant metadata such as a last updated
date, name, and description. These are all useful fields when looking at the project from the outside in, when the final
product is a docker-image it becomes especially useful as we cannot see the actual content of the image before running it.
Metadata on the other hand will be exposed if configured correctly and is in many cases need for the devops
to be able
to differentiate between the images without having to run them.
As of 22.02.22
any project that does not utilize a framework or package-manager that handles versioning, defaults to
the NPM format and uses a simplified package.json
. This is the case for both WM4 and FT4.
When updating the version for a project you should start by first committing and pushing all code changes, then perform the version update, commit and push the affected files, and finally tag this commit with new version.
For all projects the version format is written as Major.Minor.Patch-Tag.Number
:
- Major: The major version of the project, should only be updated on completion of breaking changes.
- Minor: Updated with each new feature, this could introduce breaking changes.
- Path: Updated with each fix, minor changes to documentation etc., and should never introduce breaking changes.
- Tag.Number: These two should always be combined when used, and is to be appended when testing code introduced under
one of the above version changes. It can also be used to identify a version containing code that differs from the main code base. As an example here vi have the latest Database image used by PIT, this has
some customer specific SQL witch in turn is reflected in the version Tag (1.1.0-pit
). Notice that we never write the Tag.Number as00
, we just leave it blank. If the code specific to PIT in the Database image is updated the next version would be1.1.0-pit.1
, if the main codebase is patched the next version would be1.1.1-pit
.
In many cases where it is necessary to perform continuous tests of build pipelines or deployment logic we might
have to update the version number rapidly in order to trigger a new build. As this does not relate to changes in
the main code-base, we just update the Tag.Number for the project, this tells the DevOps that the code itself
has not changed but some part of the pipeline or other build-logic has. Some keywords used in tag such as alpha
or
beta
tells us that the current version is at a testing stage. The current version of the FT4 image for example
is now at 0.9.8-beta.50
, this is interpreted as the main code-base being at version 0.9.7
while the version 0.9.8
is at beta
testing stage on its 50th build. Tag names should always be lower-case.
A good rule of thumb is here to write out some documentation per repo defining possible keywords to be used as a Tag and what these tags mean. This makes it easy for any DevOps to determine what to name test images and how to interpret existing image versions.
Some commonly used tag names are:
- alpha: An early test build with high probability of failure.
- beta : A step-up from alpha when the build has matured and has a lower probability of failure.
- pipe : Marks the testing of a pipeline and does not affect the main code base.
For each container-based project there should be a update_version.sh
script in the root. This script can be called directly
to update the version of the individual projects without triggering anything else, and is also called from build.sh
when
updating the version. This script may differ in functionality based on how versioning is done for the project.
As an example is the included update_version.sh
for FT4 which follows the rules for versioning mentioned above,
as well as performing some custom logic for updating a version file within the project. This example outlines how a
version and when update should be performed:
#!/bin/bash
# The first line in any script should be a `shebang` which tells the interpreter what runner to use.
# All Indico images as of 22.02.22 includes Bash and should match the line above for all scripts.
if [ -f ".build.env" ]; then #Checks for the local .build.env file
source .build.env #Loads the local .build.env file values if it exists.
fi
for arg in "$@"; do #Itterates arguments passed to the script
case "$arg" in
*) # Interprets any argument as next_version, passing multiple arguments to the
next_version=$arg
;;
esac
done
if [ -z "$next_version" ]; then # Checks if the next version has been sat
echo " - error: You need specify a version to update it..."
exit 1
fi
# The info file in FT4 is not interest in the `Tag.Number` part of the version, so we strip this away.
echo " - Stripping version flags"
next_version=(${next_version//-/" "})
next_version="${next_version[0]}"
flags=${next_version[1]}
if [ -n "$flags" ];then
echo " - Removed: \"$flags\""
fi
# Only matches the current version system, will stripp everythng after a "-"
if ! [[ "$next_version" =~ ^[0-9]\.[0-9]\.[0-9]$ ]]; then
echo " - error: Invalid version format \"$next_version, should be \"0.1.0\""
fi
if [ ! -f "$FT4_INFO_PATH" ]; then # Checks if the info file exists at the given location in .build.env
echo " - error: no FT4 info can be found at $FT4_INFO_PATH ..."
exit 1
fi
vs_seg=(${next_version//./" "})
echo " - Updating $FT4_INFO_PATH to version $next_version"
function update_vs() { # Performers the actual version update with jq
jq \
--arg major "${vs_seg[0]}" \
--arg minor "${vs_seg[1]}" \
--arg patch "${vs_seg[2]}" \
--arg date "$(date +%F)" \
'.version.major = ($major | tonumber) | .version.minor = ($minor | tonumber) | .version.patch = ($patch | tonumber) | .version.date = $date' \
"$FT4_INFO_PATH"
}
vs=$(update_vs)
echo "$vs" > "$FT4_INFO_PATH" # Saves the new version to the info.json file
echo " - Version updated to $next_version"
exit 0
The simplified package.json
used by FT4:
{
"name": "ft4", # Lower-case public name.
"version": "0.9.8-beta.46", # The full version.
"versionDate": "2022-02-18" # Date when the version was last updated.
}
To improve readability and streamline image building and kubernetes deployment automation we should strive as much as
possible to maintain the same project structure across all container-based projects. If the project is using a package
manager or framework, chances are that the project is already following the best practices for project-structure
. Only
a few older solutions still stuck in time deviates from this.
Furthermore, the projects should contain one project and one project only. When developing for a container-based environment the final deployment does only work with one single foreground runner. This means that after the pod starts up it is tied to s single executable which it also binds its lifecycle to. Resulting in the pod stopping when the executable completes or crashes.
A well-structured project should look something like this:
From the above picture we find all configurations, external scripts and assets at root level. While all code lives in
the src
folder, this folder could be given any name as long as it hits towards containing the source code for the
project. Commonly used names are src
or app
. Following this approach increases readability and allows us to easily
locate cache folders and personal configurations or environment files.
In the root of all projects as with the example above we find the package.json
which is the default project-descriptor
and versioning file for all Indico projects per 23.02.22
. The contents of said file looks something like this:
{
"name": "dploy",
"version": "0.1.13",
"versionDate": "2022-02-23",
...
}
Here we can easily determine the version, last updated version data, and project name from the file. Internal Indico
scripts such as build.sh
utilize this to update and parse versions for the project by default.
Projects that are hosted in Kubernetes have a special folder kubernets
at project root. This is where all .yaml
files related to the running deployment are stored. For the dploy repository
the content of the kubernetes
folder looks like this:
The files in the above image is always the currently deployed files for the project in Kubernetes. As they are kept
in the project root and in the same repository as the project itself, it forces the developer to always commit
changes made to the .yaml
files to the remote. This in turn ensures that everyone is working on the same version of
the Kubernetes resources when making changes in Kubernetes. Finally, internal Indico scripts such as build.sh
utilizes this to automatically update the images used by kubernetes/deployment.yaml
when the following two flags are
sat:
# Enables kubernetes publishing
BUILD_IMAGE_KPUB=1
# Enables kubernetes deployment update with new image version
BUILD_IMAGE_KUBER=1
build.sh is a simple bash-script that is to be added into the root of any container-based Indico-project. The intention of this script is to simplify the way Docker-Images are built and published. In addition, the script also updates project versions and commits the final result for Git on a successful build.
For the script to work properly you will have to follow the above project structure and versioning guidelines. With
your container-based project structured correctly, you download and configure the .build.env
from this repo.
In addition to the dependencies mentioned below you will need a Dockerfile
in the root of the repository for
docker build
to work. This file is tested-working and included in alle container-based Indico projects as of
25.02.22
, be careful when making changes to this file, changes made to the source code should not affect the
Dockerfile
itself.
For build.sh
to work correctly you will need a couple of system known commands and the official command-line tool from
DigitalOcean doctl
. The following is the current requirements for the script:
- dotcl: Used for communicating with the DigitalOcean container-registry, follow the Indico guide for installing and configuring this here.
- docker: Required for building docker images, follow the Indico guide for installing and configuring this here
- jq: Used for parsing
json
from commandline, download this with your favorite package manager or see the official docs for installation.
The script comprises a couple commands and a lot of flags. All flags mentioned here have a matching variable in the
.build.env
so that you don't have to append all flags each time you run the script. Some values are not possible to
set through the command-line as they are the same between projects, and should therefore be sat in the .build.env
file.
Running build.sh
with the -h
flag will take you to an internal help screen which briefly explains the different
flags, see the section below. Running build.sh --help
will take you to this documentation screen in your browser.
██████╗ ██╗ ██╗██╗██╗ ██████╗
██╔══██╗██║ ██║██║██║ ██╔══██╗
██████╔╝██║ ██║██║██║ ██║ ██║
██╔══██╗██║ ██║██║██║ ██║ ██║
██████╔╝╚██████╔╝██║███████╗██████╔╝
╚═════╝ ╚═════╝ ╚═╝╚══════╝╚═════╝
© 2022 Indico Systems AS
All rights reserved
A simple helper script for building and publishing almost
all indico software. The usage of this script is the same
across all repositories, and it depends on doctl and docker
For more information, questions or documentation pleas
contact your friendly neighbourhood Martin.
Requirements:
- doctl : DigitalOcean command line interface, see install
guide at https://dploy.indico.dev/#/digitalocean
- docker: Docker runner, for windows this should be Docker-
Desktop, guide at https://dploy.indico.dev/#/docker
- jq: : Json command line parser, required to update version
: see docs to install https://stedolan.github.io/jq/
Eks: ./build.sh 0.1.5-beta.5 -p --kuber --node --kpub -y
Usage:
-h,H : Help, shows usage.
--help : Opens the full documentation page for this script at
https://dploy.indico.dev/#/build.
-y,Y : Auto accepts image overwrite and reusing existing
build version.
-p, --publish : Publishes the docker image to the do-reguistry upon
build completion
--kuber : updates kubernetes deployment file with the new version.
NB, requires yq https://mikefarah.gitbook.io/yq/
--node : Updates package json with the new version number.
NB, requires jq https://stedolan.github.io/jq/
--kpub,kpop : Applies the kubernetes deployment changes with kubectl.
This requires kubectl to be installed and configured with
the correct kubernetes cluster. Remember to sett the
correct namespace
--tag : Add and commits the updated resources post build, pushes
the commit, and finally tags and pushes this final commit
with the new version.
v, version : Displays the last built and published version name according
to this repo.
-v,--version : Displays the last built version, and just the version.
From the help section above everything except --help, --version
starting with --
has a matching env variable
in the .build.env
file, and it is recommended to set these there to ensure consistency across builds. Calling the
script with v
or version
will display the current version and last updated date, while running it with -v
or
--version
will display only the version.
If any errors should occur related to the docker build-step, or something seems fishy after the build has completed, you can run the following command to remove all previously cached docker images:
docker rmi -f $(docker images -q -a)
If the errors persist try removing the latter FROM
parts from the Dockerfile
, run the above command again, and then
try rebuilding the image.
The .build.env
file is an Indico-environment file read by build.sh
. For every container-based project using
bash.sh
you will need to create, download, or copy this into the root of the project. Included in this repository is a
preconfigured and documented version of said file, which looks as followed:
# The docker container registry to push to
BUILD_REGISTRY_NAME="registry.digitalocean.com/indico"
# The name of the image
BUILD_IMAGE_NAME=indico-dploy
# If present the build script will override, and change the kubernetes context
# before applying any resources, the scrit will then shift back to the original context.
BUILD_KUBE_CONTEXT=dploy
# Git commit message, the default message to show with the commit published trough --tag
BUILD_GIT_MESSAGE="built new version"
# Default for all flags are 1 = enabled, and 0 = disabled
# Enables kubernetes publishing
BUILD_IMAGE_KPUB=1
# Enables kubernetes deployment update with new image version
BUILD_IMAGE_KUBER=1
# Enables node version update in package.json
BUILD_IMAGE_NODE=1
# Enables publishing of image to sat registry
BUILD_IMAGE_PUBLISH=1
# Enables git commit and tagging after updated versions
BUILD_GIT_TAG=1
# Enables auto-accept for all yes/no questions
BUILD_ALLOW=1
# The location of package.json or version file of same format, usually "./package.json"
BUILD_PROJECT_FILE="./package.json"
# Following two variables are commands, these can be a path to another bash script or and inline command.
# Additional version update command. The command specified will be called with the rsolved version as an argument.
BUILD_ADDITIONAL_VERSION=./update_version.sh
# Additional build commands, this is where you would insert "yarn build" etc. Leave empty for none.
BUILD_EXEC="yarn install && yarn build"
This is the container-registry for where to push a built docker image to on a successful build. This is required
regardless of the flag BUILD_IMAGE_PUBLISH
being sat or not, as docker needs this information in order to tag the
images correctly.
The actual shorthand name of the image, this should be kept as short as possible and not contain any special characters
except for -
. As of 25.02.22
these names are ft4 for FT4, wm4 for WM4, and vault for Vault. For the majority
of projects this is the only value you will need to update. Be careful to remember doing this with each new project
as giving this the wrong name will result in the pushed image overwriting another image.
The context name to use with kubectl
when performing an auto-update of a related kubernetes deployment. This is only
relevant for projects following the projects structure mentioned above, and has a valid /kubernetes/deployment.yaml
file at the root of the project. This field is only read when the flags for updating Kubernetes resources
BUILD_IMAGE_KUBER
, or publishing Kubernetes resources BUILD_IMAGE_KPUB
is sat. If you are going to use this
have a quick look at the Kubernetes guide first.
This is the commit message used when the flag BUILD_GIT_TAG
is enabled. This can only be a string.
Enables publishing with the Kubernets context sat with BUILD_KUBE_CONTEXT
. This is tightly tied to the resource
being published having actually been updated, and should be sat to 0
unless BUILD_IMAGE_KUBER
is sat. Requires
the project to have a /kubernetes/deployment.yaml
file.
When enabled it will update the /kubernetes/deployment.yaml
file with the newly built version on a successful build.
As mentioned above, this requires the project to follow the guidelines for good project-structure for this to work.
It is recommended to keep this flag sat to 1
if the project has a /kubernetes/deployment.yaml
file, in order to keep
the internal Kubernetes resources up to date with the last built version.
When enabled will try to update a node-like version file following the general structure of an NPM-package.json. The
file path to the file updated is unless specified sat to ./package.json
, but can be overwritten by changing the
BUILD_PROJECT_FILE
value.
With this enabled, build.sh
will on a successful build publish the resulting docker image to the container-registry
specified with BUILD_REGISTRY_NAME
.
Enabling this will on a successful build commit the changed files with the message sat to BUILD_GIT_MESSAGE
. Then it
will push this commit to the repo, and finally create and publish a Git-tag at the same point with the new version
name. When using this, remember to always commit and push all code related changes, see the Git guide
for best precises around this. This is highly recommended using as it ensures that there is always a versioned
Git-tag matching the image-version-name in the git history.
Pushing new tags and commits to server will not succeed as long as there is warning present in git. This is default behaviour for Git and should not be changed, but rather fixed or ignored before committing and running
build.sh
. Forbuild.sh
there is no problem in push step failing as the commits and tags are stored in git locally, you can go back to your repo, resolve the git issues, and then push with the regular push command andgit push --tags
. This will cause all local commits and tags to be pushed to the server.
Enabling this bypasses all prompts build.sh
should give, and auto-accepts everything. Setting this to 1
is the
same as running ./build.sh
the -y
flag, eks: ./build.sh 0.1.1 -y
.
This is the location of the NPM-package.json like file holding the project info and version. When not present in the
.build.env
file, the BUILD_PROJECT_FILE
variable will default to ./package.json
. This is also the default location
from NPM and good project structure guid above for said file.
This could be a command or path to an external script that gets called by build.sh
. If not present this step will be
skipped in build.sh
. Use this to update addition versioning files within the source code on a new build. In this
repository there is included two examples of this, one for dploy update_version.sh
, and one for ft4 update_version_ft4.sh
The newly resolved version name will be passed as an argument to the script or command specified here.
This is the pre docker build step, and is called after all version updating is done, both node and additional, but before
the docker image is built. This allows you to run commands such as yarn build
before the docker build starts with the
new version name updated. Should the command or script placed here fail with an error >= 1
will cause the build.sh
to stop and exit with error code 1, when BUIL_EXEC
is specified it will have to succeed for build.sh
to complete.
If you do not wish to use BUILD_EXEC
, then remove it completely in the project .build.env
.