rog-golang-buddies/golang-template-repository

Kickstarter repository for a golang service

golang-template-repository's Issues

Investigate feasibility of super linter GH workflow

The idea here is to basically have one workflow internally call the static checks required for a commit to a repo.

Ex: If I make a change only to a json file, I don't need to run a go static check.

This is done by a GH action called super linter, which does all of this internally after getting triggered during the action config specified. More info here:

https://github.com/github/super-linter

An example of this (check commit history):

https://github.com/pallasite99/customer_risk_analysis_api

This will make our CI workflow simpler and only run what is required, instead of POs commenting out or deleting unneeded workflows manually.

Align workflow triggers

All workflows do not have a consistent trigger mechanism. The following workflows should have a consistent trigger and if not, should be documented via a comment on why it is different. Alignment on this would also set the defaults for future workflows we setup.

name: SonarCloud
on:
  push:
    branches:
      - main
  pull_request_target:
    types: [opened, synchronize, reopened]
...

name: Gitleaks
on: [pull_request, push, workflow_dispatch]
...

name: Go-Test
on:
  [pull_request, push, workflow_dispatch]
...

name: golangci-lint
on:
  push:
  pull_request: 
...

The following seems obvious but added here for completeness.

name: gen-docs
on:
  push:
    branches:
      - main
...

name: release
on:
  push:
    branches:
      - main
 ...

Add SonarCloud documentation

SonarCloud CI usage docs must be added to https://github.com/rog-golang-buddies/golang-template-repository/blob/main/docs/continuous-integration/sonarqube.md.

The filename must also be updated from sonarqube.md to sonarcloud.md.

Build error: "undeclared name: any"

I am having a build error here: https://github.com/rog-golang-buddies/go-automatic-apps/runs/7680810060?check_suite_focus=true

The log is here:

  Error: level=error msg="Running error: buildir: analysis skipped: errors in package: [/home/runner/work/go-automatic-apps/go-automatic-apps/server/httpd/common.go:10:56: undeclared name: any]"

I found that the solution is to update Go to 1.18 or later: https://stackoverflow.com/questions/71487498/why-do-i-get-undeclared-name-any-requires-version-go1-18-or-later-when-usin

Keeping a repository updated with template repo changes

It is expected that this template repo will undergo continuous improvement changes and teams should be able to keep their repos updated as this happens. There is however some confusion around how teams are expected to keep their repositories updated in the most seamless way.

It is also evident that there some are cases with undesirable effects:

When you create a new branch which includes the template repo git commits and it has template repo references with the issue number, Github will associate the commit to the matching project repo issues (if it exists with the same issue number).

Example:

Acceptance Criteria

• Provide easy instructions to raise a PR with the latest changes from the template repo
• Ensure only changes from main branch or a tag is fetched if the template repo is added as another origin to the project repo
• The PR branch pushed to Github should not include the template repo commit history (to resolve the issue described above)

Dockerfile require statically building the binary

Adding CGO_ENABLED=0 as a sensible default to produce a statically-linked binary.

This setting is also used as the default for the distroless image:
https://github.com/GoogleContainerTools/distroless/blob/main/examples/go/Dockerfile

Further Context

cgo is enabled by default. In the repo Dockerfile, the dynamic linking occurs on golang:1.18 image and runs on gcr.io/distroless/static image where the libraries don't exist. This can cause it to fail.

Add .dockerignore file

The current Dockerfile does a copy all via COPY . . without any filtering on what is in the working directory. We should add a .dockerignore file to eliminate bloat. The current landscape has many things that can be ignored:

.
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── bin
│   └── app
├── cmd
│   └── main.go
├── coverage.out
├── docs
│   ├── continuous-integration
│   │   ├── gitleaks.md
│   │   ├── golangci-lint.md
│   │   ├── goreleaser.md
│   │   ├── img
│   │   │   └── mkdocs-gh-pages.png
│   │   ├── mkdocs-material.md
│   │   ├── pre-commit.md
│   │   └── semantic-release.md
│   ├── index.md
│   └── quickstart.md
├── go.mod
└── mkdocs.yml

Acceptance Criteria

1. Add a .dockerignore file to ignore docs, binaries, test artefacts, github actions etc.

Add go tests

A workflow is required to run go test on events .g: on: [pull_request, push, workflow_dispatch]

Evaluate Task as an alternative to Make

Documentation suggests that it is an easier to use alternative to GNU Make. It also better fits within the Go ecosystem.

https://taskfile.dev/

Release should trigger a binary build

so my limited knowledge of gitlab actions tells me that this should have created a binary:

name: SLSA Go releaser
on:
  workflow_dispatch:
  release:
    types: [created]

from https://github.com/rog-golang-buddies/golang-template-repository/blob/main/.github/workflows/go-ossf-slsa3-publish.yml#L17

there is a release but this workflow didn't get triggered. I suspect for this to fire, the release type needs to be published as shown below:

on:
  release:
    types: [published]

Without testing I can't be sure.

On a slightly different note, I'm also not familiar with SLSA Go releaser and that makes me hesitant to provide it as a solution to our user base, as we should have some level of authoritative knowledge to support it. In contrast goreleaser appears to be easy to adopt, well documented and is widely used (https://goreleaser.com/users/).

From a security posture perspective slsa-go might be well worth exploring but I wonder if it is suitable for our first release? perhaps it should be an improvement story. Happy to be corrected but this is my sentiment atm.

CC: @shukra-in-spirit , @pallasite99

Add semantic-release docs and call out caveats

semantic-release needs to be documented well. Specially the case where the first commit to master (as it is configured to act on master in our repo) will create release 1.0.0. Their convention varies as they claim 0.1.0 is not well defined and npm starts versioning at 1.0.0. However they provide pre-release guidelines:

https://github.com/semantic-release/semantic-release/blob/master/docs/recipes/release-workflow/pre-releases.md

We need to make this explicitly clear to developers as this first release as 1.0.0 can come as a surprise. Ideally a working example should be provided.

The blog post they reference:
https://blog.greenkeeper.io/introduction-to-semver-d272990c44f2

Additionally, provide them with instructions for setting up tooling to follow commit message standards via https://commitizen-tools.github.io/commitizen/

Add Gitleaks Github Action

• Add the gitleaks github action
• Add necessary description to https://github.com/rog-golang-buddies/README

Explore act, the local gh actions

To execute locally gh actions, we can investigate this option https://github.com/nektos/act, there are several options including --dry-run

Add Semantic Release

1. Add the semantics release github action
2. Add necessary description to https://github.com/rog-golang-buddies/README

Dockerfile

Create makefile

• Create a makefile with multi service support
• Publish necessary certificates

Sane default for golanglint-ci

As per https://golangci-lint.run/usage/quick-start/ golanglint-ci enables a limited set of defaults. The disabled list contains some useful linters that perhaps should be enabled as it provides greater linting coverage. IMO some examples:

• https://revive.run/
• https://github.com/butuzov/ireturn
• https://github.com/ssgreg/nlreturn
• https://github.com/Djarvur/go-err113
• etc.

These could still be perceived as subjective so some consideration must be given to what is objectively useful. It is also best to consider adding these at the start of the project rather than midway as a custom/disparate styles may have proliferated through the code base and fixing the linting issues become cumbersome.

Acceptance Criteria

• Assess linters that would be highly beneficial
• Options to disable certain rules should exist (if deemed not applicable)
• Add .golangci.yml file with enabled linters
• Update docs

When a PR is open the golangci-lint job is running twice

Proposal for fix:
Only have the job run on pull_request into branch main as it doesn't really matter if the lint fails until it is trying to be merged into main

Add a way to cross-compile builds for multiple OS and Architectures

since we don't know which user is using which OS, then executing go build or make build go will build the binary for that specific OS and Architecture. to avoid that and cross-compile the code we can use GOOS and GOARCH environment variables inside the MAKEFILE.

examples

Windows 64-bit:
GOOS=windows GOARCH=amd64 go build -o bin/app-amd64.exe app.go

Windows 32-bit:
GOOS=windows GOARCH=386 go build -o bin/app-386.exe app.go

use go:generate and go:embed instead of ldflags

more info is available here: https://levelup.gitconnected.com/a-better-way-than-ldflags-to-add-a-build-version-to-your-go-binaries-2258ce419d2d

Explore the option of buildpacks instead of Dockerfile

For many use cases buildpacks are often a better alternative for these reasons:

• all the images generated are OCI compliant
• Avoid a lot of complexity
• Builds a container image right from source code

Documentation here

Prepare for first major release

• make sure docs are updated
• there is a quickstart guide which is comprehensive
• This is a test ISSUE FOR THE BOT

Add a PR template

Include a PR template so we have consistent process to describe the PR and include what we would generally expect to be done before raising the PR.

A sample template has been added below.

As per https://github.com/rog-golang-buddies/golang-template-repository/wiki/Platform-Team-Onboarding-Guide use github as the commit scope.

Example

## Description

_Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change._

Fixes # (issue)

## Type of change

Please delete options that are not relevant.

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] This change requires a documentation update

## How Has This Been Tested?

_Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration_

## Checklist:

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have checked my code/docs and corrected any misspellings

Add Goreleaser Github Action

Introduce the go-releaser github action to deliver go packages/binaries as a part of the release workflow

Add dependabot

• Add dependabot
• Configure for weekly update scans
• Add necessary description to https://github.com/rog-golang-buddies/README

Proposal for docs in repo

Summary

Keep documentation within the repo instead of a remote wiki or Github wiki.

Rationale

From a developer experience perspective, it is easier to clone and have everything you need to reference locally rather than having to externally reference something.

This minimises context switching and aids the practice of keeping documentation updated as it is closer to the codebase.

Solution Proposal

Leverage automatic documentation generation via https://squidfunk.github.io/mkdocs-material.

This integrates well with Github, providing automatic docs for your repo at <username>.github.io/<repository>.

Sample structure:

docs
├── ci
│   ├── gitleaks.md
│   ├── golang-ci.md
│   └── semantic-release.md
└── starter.md # docs on how to get started with this repo

Example:

https://cli.urfave.org/

Generated from ./docs directory:
https://github.com/urfave/cli/tree/main/docs

SPIKE: Pre-commit hooks

• Explore on pre commit hooks
• Document your findings on https://github.com/rog-golang-buddies/README
• Create issues on possible implementation

How to build and release a CLI?

The project https://github.com/rog-golang-buddies/go-automatic-apps will have a CLI executable available to run commands to create a new app, new model, and more operations.

Is this being contemplated in this template? If so, how it's done? How will the users install the executable?

Add local tooling info to starter docs for day 1

For day 1 requirements add purpose of tool including instructions for installing/configuring:

1. pre-commit so the hooks run when they begin their git commits. Instructions should also contain how to have pre-commit automatically run when a repo is cloned (see pre-commit docs).

2. commitizen to provide them with tooling to follow commit message standards via

1 & 2 can be installed via python pip packages.

3. How to get latest changes from the template repo

Improvements to Dockerfile

The existing Dockerfile is good for a starter project, but for a template repo we should follow some best practices.
I am including the below code for reference.

FROM golang:1.18 as builder

RUN apk update && apk upgrade && \
    apk add --no-cache make

WORKDIR /src
COPY . .

# get packages and build
RUN make tidy
RUN make build

# Using a distroless image from https://github.com/GoogleContainerTools/distroless
FROM gcr.io/distroless/static:nonroot

COPY --from=build /app/bin/app /

USER 65532:65532
CMD["/app"]

ENV PORT=8080
EXPOSE $PORT

Add docker build and run targets to Makefile

We should add make docker-build and make docker-run targets to the Makefile since we ship the repo with a Dockerfile.

Acceptance Criteria

• Add targets
• Provide a configurable variable to override the registry/image name

Fix Sonarqube scan failure "Set the SONAR_TOKEN env variable."

The Sonarqube job appears to be failing consistently in most PRs. I don't have access to check the secret settings but I do think this only occurs when a PR is raised from a forked repo.

Example:
https://github.com/rog-golang-buddies/golang-template-repository/runs/7134639006?check_suite_focus=true

Refresh README

The currently README is out of date and should be updated to give an intro into how to navigate this repo. Namely, it should provide:

1. A basic README structure since the bulk of the content will be hosted in gh-pages
2. A link to this repo's gh-pages site so that users are aware of how to get started.

Downgrade v1.0.0 release

The release v1.0.0 was automatically tagged via the semantic-release workflow when it was merged to master (semantic-release default start tag is v1.0.0). v1.0.0 represents a stable production ready release and this repo isn't there yet. We should downgrade the release to v0.1.0 as it better represents the state of the repo.

Actions required:

1. Delete v1.0.0 release and tag
2. Create a new tag and release v0.1.0 via UI

Add support for upload application docker image to dockerhub

It is good to have a workflow which publishes our application bundles into a docker image in dockerhub. On every successful release we can have the image pushed to public repository.

Refs:

• dockerhub
• Publishing docker images

Makefile does not work in Windows

The Makefile contains a call to the "sed" command and it is not available on Windows.

golang-template-repository/Makefile

Line 8 in d9937d1

@sed -n 's/^##//p' $(MAKEFILE_LIST) | column -t -s ':' | sed -e 's/^/ /'

Could this be done in a different way?