erolkeskiner/ci-cd-playground

This repository is intended to create an example basic Web Application and to create related CI/CD pipelines.

CI/CD Playground

This repository is intended to create a basic Flask Web Application and to create related CI/CD pipelines using Minikube, Helm, Jenkins and Terraform.

Introduction

This project walks through implementing a simple CI/CD pipeline for a Python Flask application. The following is demonstrated:

• Integrating Jenkins with a GitHub project
• Implementing a CI/CD pipeline using a Jenkinsfile
• Testing and running static code analysis, publishing related reports to the Jenkins Pipeline
• Building the Python package
• Building the Docker image using the Python package
• Publishing the Docker image to Docker Hub
• Deploying the Docker image to a K8s cluster

Prerequisites

Before you get started you'll need to have these:

• GitHub Account
• Docker Hub Account or any other public/private Docker Image Registry Account
• A local or remote Jenkins server. In this project, a local installation used. Refer here for the local installation instructions and the configuration for this project
• A local or remote K8s cluster. In this project, Minikube used for a local K8s cluster installation. If you will also use the Minikube, you'll need a container or virtual machine manager, such as: Docker, Hyperkit, Hyper-V, KVM, Parallels, Podman, VirtualBox, or VMWare
• Python Version >= 3.6

Project Structure

.
├── Jenkinsfile
├── Makefile
├── README.md
├── app/
├── deploy/
│   ├── helm-chart/
│   └── terraform/
└── local-installations/
    └── jenkins/

app directory contains the source code and configuration files of the application.

deploy directory contains the Helm Chart and the Terraform configuration files to deploy the application to a K8s cluster.

local-installations directory contains related documents to refer for installing and configuring Jenkins server.

Workflow

On every pipeline execution, the code goes through the following steps:

1. Code is cloned from this repository, built, tested and analyzed for bugs and bad patterns.
2. A Docker Container Image is built then published to Docker Container Registry.
3. If all previous steps finished successfully, the pipeline is paused for the approval to continue to the deployment.
4. If approved, the container image is deployed in a fresh new container in related K8s namespace.

Targeted release version for the Python package and Docker image is set in the target-version.json file in the app directory. Three tags were used as suffixes to separate the packages: dev (development), rc (release candidate), and no suffix (release).

• Any pipeline running from not main branches (a.k.a. feature/bug fix branches) builds the Python package and Docker image with the <target-version>-dev-<uuid> tag, then publishes the Docker image to registry and deploys it to dev namespace on the K8s cluster.
• Any pipeline running from the main branch builds the Python package and Docker image with the <target-version>-rc-<uuid> , then publishes the Docker image to registry and deploys it to rc namespace on the K8s cluster.
• Any pipeline running from git tag builds the Python package and Docker image with the <target-version> tag, then publishes the Docker image to registry and deploys it to prod namespace on the K8s cluster.

Thoughts and Future Works

For a healthier SDLC:

• Make your developments in feature branches.
• Create short-lived branches.
• Commit often.
• Run the pipeline to see failures fast, to test the deployment on K8s.
• Prevent merging unhealthy branches to main branch.

Expand the release version identification with community standards.

Add more stages between development, release candidate, and release. This is not necessary since this is a personal and one-person project. But, with a growing team, it would be better to have more controls before merging the feature branches to the main branch.

The CI/CD pipeline covered in this project is only for one application. Add some other units such as a database or another service that the app will communicate with and implement a mechanism for integration tests.

Test the Terraform configurations using a tool such as kitchen

ci-cd-playground's Issues

Gracefully shutdown the application server

Shutdown timeout is 30 seconds
Ignore new requests
Pod is killed after 30 seconds

Structured logging

Request
HTTP status

Local Terraform state file

The pipeline needs to import the Terraform state before executing the deployment.
This is causing state problems if the target namespace and Helm release does not already exist or already created but not with Terraform.
Since this is a local development environment, use a directory on the local machine for Terraform state files.

K8s preparations

Create '/healtz' endpoint(s) for liveness and readiness probes

Separate /versionz endpoint to another container

Create another Flask app to serve /versionz endpoint to run in another container next to main app.
Create another container image to use as a reverse proxy in front of the main app and versionz app.

Create Dockerfile for the Web App

Use alpine as the base image

Create a mechanism to create unique versions for each Jenkins Build pipeline jobs

tags are not included
main branch: <target_version>-rc-<unique_id>-<#build>
not main branch: <target_version>-rc-<unique_id>-<#build>

Create a Jenkinsfile

Run tests and/or code analysis
Build Docker image
Publish Docker image
Deploy the published image to minikube based on branches and tags using created Helm Chart

Create a basic Flask Web App

Create a basic Flask Web App

Create Terraform configurations to deploy the application to Minikube using Helm Chart

Harden the Docker image for production security

Apply Docker and Flask security best practices
Links below may be good guidelines:

https://www.toptal.com/flask/flask-production-recipes#:~:text=Although%20Flask%20has%20a%20built,and%20proxying%20request%20with%20Nginx.
https://gist.github.com/jumanjiman/f9d3db977846c163df12
https://pythonspeed.com/products/productionquickstart/

Prepare for Release Management: Package the application

Transform the pattern for package creation
Create versioned whl and/or tgz files

Create a README.md file

Create a README.md file for Makefile usage, project structure and CI/CD workflow

Create a Helm Chart for the Web App

Enable/disable a sidecar container in the deployment with a flag in the values file
If sidecar is enabled deploy it in front of the Web App container as a proxy

Configure gunicorn to log date in ISO format

Format example:
2021-01-27T01:14:38.084968

Create a Makefile for the Web App

Clean workspace
Create virtualenv
Resolve dependencies
Refresh dependencies
Build docker image(s)
Run unit tests
Run the Web App locally with Python command (use the command-line argument for the listening port (defaults to 8080))
Run the Web App locally in a Docker container (use the command-line argument for the listening port (defaults to 8080))
Run code style and standards analysis with flake8 (or another linter)

Install minikube locally

(optional) with a shell script

Version info endpoint

Create '/versionz' endpoint that returns a JSON with query time, git hash and name of the project

Install Jenkins locally

(optional) installation on minikube