thockin/go-build-template

A Makefile/Dockerfile example for Go projects.

Go app template build environment

This is a skeleton project for a Go application, which captures the best build techniques I have learned to date. It uses a Makefile to drive the build (the universal API to software projects) and a Dockerfile to build a docker image.

This has only been tested on Linux, and depends on Docker buildx to build.

Customizing it

To use this, simply copy this repo and make the following changes:

Makefile:

• change BINS to your binary name(s)
• replace cmd/myapp-* with one directory for each of your BINS
• change REGISTRY to the Docker registry you want to use
• choose a strategy for VERSION values - git tags or manual
• maybe change ALL_PLATFORMS
• maybe change BASE_IMAGE (it must be a manifest-list with support for all platforms in ALL_PLATFORMS)

Dockerfile.in:

• maybe change or remove the USER if you need

go.mod:

• change module name to the one you want to use

Go Modules

This assumes the use of go modules (which is the default for all Go builds as of Go 1.13).

Dependencies

This includes go-licenses and golangci-lint, but they are kept in the tools sub-module. If you don't want those (or their dependencies, they can be removed.

Building

Run make or make build to compile your app. This will use docker buildx (which you need to have installed) to build your app, with the current directory volume-mounted into place. This will store incremental state for the fastest possible build. Run make all-build to build for all architectures.

Run make container to build the container image. It will calculate the image tag based on the most recent git tag, and whether the repo is "dirty" since that tag (see make version). Run make all-container to build containers for all supported architectures.

Run make push to push the container image to REGISTRY. Run make all-push to push the container images for all architectures.

Run make manifest-list to build and push all containers for all architectures, and then publish a manifest-list for them.

Run make clean to clean up.

Run make help to get a list of available targets.

Testing

Run make test and make lint to run tests and linters, respectively. Like building, this will use docker to execute.

The golangci-lint tool looks for configuration in .golangci.yaml. If that file is not provided, it will use its own built-in defaults.

go-build-template's Issues

Reduce the app build time by avoiding downloading the go pkgs everytime we build

Every time we issue make build command, we enter a clean build environment & so the go pkgs (guided by go.mod) will be downloaded every single time. See the example below

scratch/go-build-template (master)$ make build
# building for darwin/amd64
go: downloading golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914
go: downloading google.golang.org/api v0.51.0
go: downloading cloud.google.com/go v0.87.0
go: downloading google.golang.org/grpc v1.39.0
go: downloading golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420
go: downloading github.com/googleapis/gax-go/v2 v2.0.5
go: downloading go.opencensus.io v0.23.0
go: downloading github.com/golang/protobuf v1.5.2
go: downloading google.golang.org/genproto v0.0.0-20210716133855-ce7ef5c701ea
go: downloading google.golang.org/protobuf v1.27.1
go: downloading golang.org/x/text v0.3.6
go: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
-ne binary: bin/darwin_amd64/myapp-1


-ne binary: bin/darwin_amd64/myapp-2


scratch/go-build-template (master)$ git diff go.mod
diff --git a/go.mod b/go.mod
index 00ae21c..76833d8 100644
--- a/go.mod
+++ b/go.mod
@@ -2,4 +2,7 @@ module github.com/thockin/go-build-template

 go 1.15

-require github.com/google/go-licenses v0.0.0-20210623184300-9376cf9847a0 // indirect
+require (
+       golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914
+       google.golang.org/api v0.51.0
+)
scratch/go-build-template (master)$

If we can save the go: downloaded packages, in a persistent place inside the workspace, we can avoid downloading the same packages again & again, thereby saving build time.

I would like to open a PR on this. Please let me know your thoughts. Thanks.

manifest-tool doesn't build

7d5bebf on go version go1.19-pre4, running make manifest-list:

../../../go/pkg/mod/github.com/deislabs/[email protected]/pkg/auth/docker/login.go:7:2: missing go.sum entry for module providing package github.com/docker/docker/api/types (imported by github.com/deislabs/oras/pkg/auth/docker); to add:
	go get github.com/deislabs/oras/pkg/auth/[email protected]
../../../go/pkg/mod/github.com/estesp/manifest-tool/[email protected]/cmd/manifest-tool/main.go:6:2: missing go.sum entry for module providing package github.com/docker/docker/cli/config (imported by github.com/estesp/manifest-tool/v2/cmd/manifest-tool); to add:
	go get github.com/estesp/manifest-tool/v2/cmd/[email protected]
../../../go/pkg/mod/github.com/docker/[email protected]+incompatible/cli/config/config.go:14:2: missing go.sum entry for module providing package github.com/docker/docker/pkg/homedir (imported by github.com/docker/cli/cli/config); to add:
	go get github.com/docker/cli/cli/[email protected]+incompatible
../../../go/pkg/mod/github.com/deislabs/[email protected]/pkg/auth/docker/login.go:8:2: missing go.sum entry for module providing package github.com/docker/docker/registry (imported by github.com/deislabs/oras/pkg/auth/docker); to add:
	go get github.com/deislabs/oras/pkg/auth/[email protected]
make: *** [Makefile:257: manifest-list] Error 1

Dependency management seems broken?

Hi Tim:

Thanks for putting together this excellent template. I'm having issues with dependency management and would appreciate any pointers.

The build image does not appear to have git installed, and doesn't seem to be correctly fetching dependencies. I could not figure out how this is done from reading through the Makefile. I would greatly appreciate any pointers you can provide.

Thanks!

Unable to run output container on Mac

Hi @thockin

I cloned a fresh repo then built the container with make container but failed to run that output container.
Did you experience that issue? Please kindly share me your suggestion.

I really appreciate your time, thank you!

My mac, docker version and the terminal output are below.

❯ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.5
BuildVersion:	19F101
❯ docker --version
Docker version 19.03.8, build afacb8b

~/Workspace/sandbox ❯ git clone https://github.com/thockin/go-build-template.git                                                                               at 23:12:31
Cloning into 'go-build-template'...
remote: Enumerating objects: 17, done.
remote: Counting objects: 100% (17/17), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 181 (delta 7), reused 11 (delta 4), pack-reused 164
Receiving objects: 100% (181/181), 40.36 KiB | 172.00 KiB/s, done.
Resolving deltas: 100% (95/95), done.


~/Workspace/sandbox ❯ cd go-build-template
~/Workspace/sandbox/go-build-template on master ❯ make container
Unable to find image 'golang:1.14-alpine' locally
1.14-alpine: Pulling from library/golang
df20fa9351a1: Pull complete 
ed8968b2872e: Pull complete 
a92cc7c5fd73: Pull complete 
9e0cccf56431: Pull complete 
cbe0275821fc: Pull complete 
Digest: sha256:6042b9cfb4eb303f3bdcbfeaba79b45130d170939318de85ac5b9508cb6f0f7e
Status: Downloaded newer image for golang:1.14-alpine
binary: bin/darwin_amd64/myapp-1
Sending build context to Docker daemon  14.57MB
Step 1/4 : FROM gcr.io/distroless/static
latest: Pulling from distroless/static
9ff2acc3204b: Pull complete 
Digest: sha256:c6d5981545ce1406d33e61434c61e9452dad93ecd8397c41e89036ef977a88f4
Status: Downloaded newer image for gcr.io/distroless/static:latest
 ---> e4d2a899b1bf
Step 2/4 : ADD bin/darwin_amd64/myapp-1 /myapp-1
 ---> d98c16a13428
Step 3/4 : USER 65535:65535
 ---> Running in 3cf5223d04ec
Removing intermediate container 3cf5223d04ec
 ---> 62e3e49cd313
Step 4/4 : ENTRYPOINT ["/myapp-1"]
 ---> Running in 1176246fabaa
Removing intermediate container 1176246fabaa
 ---> 6f6ee4fc161e
Successfully built 6f6ee4fc161e
Successfully tagged example.com/myapp-1:afae457__darwin_amd64

binary: bin/darwin_amd64/myapp-2
Sending build context to Docker daemon  14.58MB
Step 1/4 : FROM gcr.io/distroless/static
 ---> e4d2a899b1bf
Step 2/4 : ADD bin/darwin_amd64/myapp-2 /myapp-2
 ---> aad5923b077d
Step 3/4 : USER 65535:65535
 ---> Running in ccf0c5c7bd1b
Removing intermediate container ccf0c5c7bd1b
 ---> c9c5fd527e78
Step 4/4 : ENTRYPOINT ["/myapp-2"]
 ---> Running in 3fe617aba61b
Removing intermediate container 3fe617aba61b
 ---> 07dcd90f456b
Successfully built 07dcd90f456b
Successfully tagged example.com/myapp-2:afae457__darwin_amd64

container: example.com/myapp-1:afae457__darwin_amd64
container: example.com/myapp-2:afae457__darwin_amd64


~/Workspace/sandbox/go-build-template on master ❯ docker run example.com/myapp-1:afae457__darwin_amd64
standard_init_linux.go:211: exec user process caused "exec format error"
~/Workspace/sandbox/go-build-template on master ❯       

multiple BINs

If there were multiple BINs under cmd/, how should I make ?

make all-container failure

When building all containers on a OSX host

$ make all-container
building: bin/amd64/myapp
Sending build context to Docker daemon  7.117MB
Step 1/5 : FROM alpine
 ---> 76da55c8019d
Step 2/5 : MAINTAINER Tim Hockin <[email protected]>
 ---> Using cache
 ---> a48cba04c826
Step 3/5 : ADD bin/amd64/myapp /myapp
 ---> Using cache
 ---> 98797f6223ad
Step 4/5 : USER nobody:nobody
 ---> Using cache
 ---> df8607d69675
Step 5/5 : ENTRYPOINT /myapp
 ---> Using cache
 ---> 0a02c9f41938
Successfully built 0a02c9f41938
Successfully tagged thockin/myapp-amd64:1101a1d
container: thockin/myapp-amd64:1101a1d
building: bin/arm/myapp
Sending build context to Docker daemon  13.56MB
Step 1/5 : FROM armel/busybox
 ---> 7ab410cdaa53
Step 2/5 : MAINTAINER Tim Hockin <[email protected]>
 ---> Using cache
 ---> 122edf1d71f5
Step 3/5 : ADD bin/arm/myapp /myapp
ADD failed: stat /var/lib/docker/tmp/docker-builder309680552/bin/arm/myapp: no such file or directory
make[1]: *** [.container-thockin_myapp-arm-1101a1d] Error 1
make: *** [container-arm] Error 2

Generated binaries:

$ tree bin
bin
├── amd64
│   ├── darwin_amd64
│   └── myapp
└── arm
    ├── darwin_arm
    └── linux_arm
        └── myapp

Notice the non amd64 architecture options have an additional level of indirection where the binary is stored.

The Dockerfile template expects ADD bin/ARG_ARCH/ARG_BIN /ARG_BIN

non amd64 generation is bin/ARG_ARCH/OS_ARCH/ARG_BIN note the OS_ARCH folder

Nested cmd directories support

cmd/acquiring/server
       /acquiring/migrate

Fails with:

$ BINS=acquiring/api make build

# building for darwin/arm64
binary: bin/darwin_arm64/acquiring/server  mv: rename .go/bin/darwin_arm64/acquiring/server to bin/darwin_arm64/acquiring/server: No such file or directory
make: *** [.go/bin/darwin_arm64/acquiring/server.stamp] Error 1

Multiple registry support

Some open repos like to push to multiple container registries.
Like we are now going to support GCE container registry, recently released AWS public container registry and dockerhub.

This needs to be done multiple times at present for each registry:
make REGISTRY=public.ecr.aws/abc push

verify-boilerplate.sh in kubernetes/contrib failed for extra "#".

For shell scripts in build folder(build.sh and test.sh).

May replace with a blank line?

Support cross-compiling to other OS

Have you given any consideration to cross-compilation for building for other OS's?

A sketch (I'm no Makefile expert) for how this could work in #29

Just a very very very minor typo?

it should be if? Or, iff represents if and only if?
https://github.com/thockin/go-build-template/blob/master/Makefile#L92

Pulling Images Across Boundaries

I am grateful for this repo, as it has accelerated my understanding. I do have a question regarding versioning. How does the "git" tagging actually work in the real world, in terms of image pulling etc, when you are trying to grind through a container to container test? (each container-image has a different revision, which you are simultaneously improving). Are you just using a local docker repo and referencing latest?

I am definitely missing something in the workflow on how to annotate my k8s yaml to accept a particular branch versus a hard coded version.

Failure when the BIN name has underscores

$ BINS=grpc_go_tmpl_server make build
# building for darwin/amd64
binary: bin/darwin_amd64/grpc_go_tmpl_clientmv: rename .go/bin/darwin_amd64/grpc_go_tmpl_client to bin/darwin_amd64/grpc_go_tmpl_client: No such file or directory

go 1.10 gives warning about build cache

Having copied this repo's contents into my project, when I run make, I get

ncc@mal:~/src/github.com/ncabatoff/spent$ make
building: bin/amd64/spent
go: disabling cache (/.cache/go-build) due to initialization failure: mkdir /.cache: permission denied

The only impact appears to be that it breaks 'go test':

ncc@mal:~/src/github.com/ncabatoff/spent$ make test
Running tests:
go: disabling cache (/.cache/go-build) due to initialization failure: mkdir /.cache: permission denied
go: disabling cache (/.cache/go-build) due to initialization failure: mkdir /.cache: permission denied
?   	github.com/ncabatoff/spent/cmd/spent	[no test files]
?   	github.com/ncabatoff/spent/pkg/version	[no test files]

Checking gofmt: PASS

Checking go vet: FAIL
go: disabling cache (/.cache/go-build) due to initialization failure: mkdir /.cache: permission denied

Makefile:148: recipe for target 'test' failed
make: *** [test] Error 1

I was able to work around this by adding -e "GOCACHE=/tmp/gocache" to the test docker invocation in the Makefile.

License scanning (thus container building) fails in non-vendor-ed repo

Repro:

Take this repo:

[hi on] jingyuanliang@jingyuanliang:/tmp$ git clone https://github.com/thockin/go-build-template.git
Cloning into 'go-build-template'...
remote: Enumerating objects: 526, done.
remote: Counting objects: 100% (320/320), done.
remote: Compressing objects: 100% (101/101), done.
remote: Total 526 (delta 218), reused 265 (delta 204), pack-reused 206
Receiving objects: 100% (526/526), 305.65 KiB | 3.97 MiB/s, done.
Resolving deltas: 100% (304/304), done.
[hi on] jingyuanliang@jingyuanliang:/tmp$ cd go-build-template/

Add reference to a third-party package; tidy, but do not vendor:

[hi on] jingyuanliang@jingyuanliang:/tmp/go-build-template$ vim cmd/myapp-1/main.go 
[hi on] jingyuanliang@jingyuanliang:/tmp/go-build-template$ git diff
diff --git a/cmd/myapp-1/main.go b/cmd/myapp-1/main.go
index d5d2cc7..cdcf8f9 100644
--- a/cmd/myapp-1/main.go
+++ b/cmd/myapp-1/main.go
@@ -21,6 +21,8 @@ import (
        "log"
 
        "github.com/thockin/go-build-template/pkg/version"
+
+       _ "k8s.io/apimachinery/pkg/api/errors"
 )
 
 func main() {
[hi on] jingyuanliang@jingyuanliang:/tmp/go-build-template$ go mod tidy
go: finding module for package k8s.io/apimachinery/pkg/api/errors
go: found k8s.io/apimachinery/pkg/api/errors in k8s.io/apimachinery v0.28.1

Now try to build the container, and it fails:

[hi on] jingyuanliang@jingyuanliang:/tmp/go-build-template$ make container
# building for linux/amd64
go: downloading k8s.io/apimachinery v0.28.1
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd
go: downloading k8s.io/klog/v2 v2.100.1
go: downloading k8s.io/utils v0.0.0-20230406110748-d93618cff8a2
go: downloading github.com/google/gofuzz v1.2.0
go: downloading golang.org/x/net v0.13.0
go: downloading gopkg.in/inf.v0 v0.9.1
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/go-logr/logr v1.2.4
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading golang.org/x/text v0.11.0
binary: bin/linux_amd64/myapp-1  
go: downloading github.com/google/go-licenses v1.6.0
go: downloading github.com/spf13/cobra v1.6.1
go: downloading github.com/otiai10/copy v1.6.0
go: downloading golang.org/x/tools v0.7.0
go: downloading github.com/google/licenseclassifier v0.0.0-20210722185704-3043a050f148
go: downloading gopkg.in/src-d/go-git.v4 v4.13.1
go: downloading golang.org/x/text v0.8.0
go: downloading go.opencensus.io v0.23.0
go: downloading k8s.io/klog/v2 v2.80.1
go: downloading golang.org/x/net v0.8.0
go: downloading golang.org/x/mod v0.9.0
go: downloading github.com/go-logr/logr v1.2.3
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading gopkg.in/src-d/go-billy.v4 v4.3.2
go: downloading golang.org/x/crypto v0.5.0
go: downloading github.com/sergi/go-diff v1.2.0
go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
go: downloading github.com/emirpasic/gods v1.12.0
go: downloading github.com/src-d/gcfg v1.4.0
go: downloading github.com/xanzy/ssh-agent v0.2.1
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd
go: downloading golang.org/x/sys v0.6.0
go: downloading gopkg.in/warnings.v0 v0.1.2
F0907 07:53:38.225951       1 main.go:77] err: exit status 1: stderr: go: could not create module cache: mkdir /go/pkg/mod: permission denied
make: *** [Makefile:226: .licenses] Error 1
[hi on] jingyuanliang@jingyuanliang:/tmp/go-build-template$ 

The reason is that without a vendor directory, go-licenses makes .go/pkg/mod, and .go/pkg is not pre-created thus owned by root (created by docker):

[hi on] jingyuanliang@jingyuanliang:/tmp/go-build-template$ ls -l .go
total 12
drwxr-x--- 3 jingyuanliang primarygroup 4096 Sep  7 07:53 bin
drwxr-x--- 5 jingyuanliang primarygroup 4096 Sep  7 07:53 cache
drwxr-xr-x 2 root          root         4096 Sep  7 07:53 pkg
[hi on] jingyuanliang@jingyuanliang:/tmp/go-build-template$ 

`make manifest-list` broken at head

/.cache/gomodcache/github.com/estesp/manifest-tool/[email protected]/cmd/manifest-tool/main.go:8:2: no required module provides package github.com/docker/docker/cli/config; to add it:
	go get github.com/docker/docker/cli/config
make: *** [Makefile:327: manifest-list] Error 1

Seems to be broken by #100.

Shell scripts use #!/bin/sh when they should use #!/bin/bash

pipefail is a bash thing.

Add lint support

We have a project based on this go-build-template and @bcheung added linter (golangci-lint) support in it: GoogleCloudPlatform/netd@046dac1. However this makes updating from the template a bit annoying. I wonder if it should be considered to include a linter in the template, either by absorbing this one or implementing a new one.

Release creation and upload assets

make create-release
make upload-assets

create the release with the change log information.

Useful script: https://gist.github.com/foca/38d82e93e32610f5241709f8d5720156

Consider migrating GO_VERSION to the standardized GOTOOLCHAIN and `toolchain` directive in go.mod

https://chat.google.com/room/AAAABbBOUj4/MT7pG_GTFWU

cc @BenTheElder

Makefile does not accommodate working directory with spaces

If the current working directory (as presented by pwd) contains spaces, the Docker volume mounting arguments fail to parse correctly. Either quote these arguments appropriately, or document that the project can't work in a directory whose name contains spaces.

I'll submit a proposed fix shortly.

fail to build on older depnednecy version

Great tool and thanks for sharing it.

I am trying to use go zeromq binding but finding errors with make file build I think because the version is pre go mod or doesn't have support for vendor mod.

require (
	github.com/zeromq/goczmq v4.1.0+incompatible
)

make

making bin/linux_amd64/dwx
build github.com/zeromq/goczmq: cannot load github.com/zeromq/goczmq: no Go source files
make: *** [Makefile:96: .go/bin/linux_amd64/dwx.stamp] Error 1

Any idea how to fix this?
Thanks.

Have an error when run a command: make container

OS: ubuntu 20.04
Architecture: amd_64

Builds for windows platform fails

Needed to release binaries for windows platforms:

• Added Windows Platform (windows/amd64)

$ git diff
diff --git a/Makefile b/Makefile
index 6285641..b3bc1d2 100644
--- a/Makefile
+++ b/Makefile
@@ -30,7 +30,7 @@ VERSION ?= $(shell git describe --tags --always --dirty)

 SRC_DIRS := cmd pkg # directories which hold app source (not vendored)

-ALL_PLATFORMS := linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/s390x
+ALL_PLATFORMS := linux/amd64 windows/amd64

 # Used internally.  Users should pass GOOS and/or GOARCH.
 OS := $(if $(GOOS),$(GOOS),$(shell go env GOOS))

• Tried building which fails. Tried making all containers, fails.

$ make all-build
binary: bin/linux_amd64/myapp-1
binary: bin/linux_amd64/myapp-2
binary: bin/windows_amd64/myapp-1
mv: rename .go/bin/windows_amd64/myapp-1 to bin/windows_amd64/myapp-1: No such file or directory
binary: bin/windows_amd64/myapp-2
mv: rename .go/bin/windows_amd64/myapp-2 to bin/windows_amd64/myapp-2: No such file or directory

$ make all-container
binary: bin/linux_amd64/myapp-1
binary: bin/linux_amd64/myapp-2
container: example.com/myapp-1:afae457-dirty__linux_amd64
container: example.com/myapp-2:afae457-dirty__linux_amd64
binary: bin/windows_amd64/myapp-1
mv: rename .go/bin/windows_amd64/myapp-1 to bin/windows_amd64/myapp-1: No such file or directory
Sending build context to Docker daemon  51.93MB
Step 1/4 : FROM gcr.io/distroless/static
 ---> e4d2a899b1bf
Step 2/4 : ADD bin/windows_amd64/myapp-1 /myapp-1
ADD failed: stat /var/lib/docker/tmp/docker-builder524817734/bin/windows_amd64/myapp-1: no such file or directory
make[1]: *** [.container-example.com_myapp-1-afae457-dirty__windows_amd64] Error 1
make: *** [container-windows_amd64] Error 2

Please suggest, if it is supported, or if there is something i am doing wrong.

make clean permission errors

Hi just started using this template, it's pretty awesome. Had to make some modifications for it work when using cgo but I'm not sure if that is causing this issue. I think it has to do with how go downloads modules.

Running make clean results permission errors when make is trying rm -rf .go/pkg/mod.
example:

rm: cannot remove '.go/pkg/mod/golang.org/x/[email protected]/sha3/testdata/keccakKats.json.deflate': Permission denied

Not sure if others are running into this problem as well

Running Docker version 20.10.8, build 3967b7d
go v1.17

The solution I came up with is to have a go-clean step

go-clean: # @HELP runs go clean -modcache to remove the modules under pkg/mod
go-clean: $(BUILD_DIRS)
	@echo "# launching a shell in the containerized build environment to run go clean -modcache"
	@docker run                                                 \
	    -ti                                                     \
	    --rm                                                    \
	    -u $$(id -u):$$(id -g)                                  \
	    -v $$(pwd):/src                                         \
	    -w /src                                                 \
	    -v $$(pwd)/.go/pkg:/go/pkg:z                            \
	    $(BUILD_IMAGE)                                          \
	    /bin/sh -c 'go clean -modcache'

Not sure if you have a better solution but I wasn't able to find anything about it. There is a thread about the permissions here: golang/go#27161

Make test: permission denied

Hello,

running test seems to be broken on my Fedora 32.

make test

/bin/sh: ./build/test.sh: Permission denied
make: *** [Makefile:209: test] Error 126

The normal make and make containers work fine.

what am I missing?

version.VERSION is written in all capital letters

Nitpick: shouldn't the VERSION const be renamed to Version to follow the CamelCase convention?

ARMv7 support

Thanks for this awesome build template

How can I make an image fit for Raspberry PI 3 B -> ARMv7?

The image has:

docker image info

"Architecture": "amd64",

docker service create ...

fail for

1/4: no suitable node (unsupported platform on 4 nodes) 
2/4: no suitable node (unsupported platform on 4 nodes) 
3/4: no suitable node (unsupported platform on 4 nodes) 
4/4: no suitable node (unsupported platform on 4 nodes) 

can I add anything to the build to make this work?

using docker run works (and container run fine) on same image but swarm seemed to do more checks.

What did I get wrong or how to fix this?
Thanks in advance

Blog Post - Makefile you need!

I am writing a blog post to convince others to use go-build-template and have a Makefile like this repo. I have used this for a long time now and like its Makefile very much. Thank you.

Few screenshots on blog - WIP

Few things I am considering

• All tasks are behind container - Consistent experience and no deps to install locally, clone, and start using. Everything is there in the build image.
• CI works the same
• Make gives the consistent API across repos for the build, test, etc...
• Image caching? Since we miss out on docker multi-stage builds, does the caching of images a concern with the template Dockerfile.in the approach
• Single Dockerfile for all deployable in a repo - I see many repos maintain tons of Dockerifle for diff environments - like dev ones need extra things installed, etc...
• Parameterized Generic Makefile

@thockin please suggest some top pointers and considerations as content that you think should be there in that post and I may be missing out.

Feature: Versioning!

Augment the boiler plate to include the git version/tag as part of the binary. This is a super common thing that is kind of hard to set up.

Duplicated runs in tests workflow

https://github.com/thockin/go-build-template/blob/master/.github/workflows/tests.yaml uses matrix go-version: [ "1.17", "1.18", "1.19" ] but this is not passed into the make command, so all runs of the tests workflow are actually the same.

Support for private repos in build and other tasks

GOPRIVATE needs to be set if there are private repos that needs to be pulled
Also, the container would need some kind of GIT_TOKEN or ~/.netrc file to be able to pull private dependencies

Pull fails at present since these are not found

$ make build

fatal: could not read Username for 'https://github.com': terminal prompts disabled1

PUBLISH tags feature: Option to release all sem version formats

We have added a publish tag feature like this here https://github.com/practo/k8s-worker-pod-autoscaler/blob/master/Makefile#L14-L17

This helps in releasing all the sem versions together.

$ make push
pushed: practodev/workerpodautoscaler:v1.0.0-19-g59d0d48
pushed: practodev/workerpodautoscaler:v1.0
pushed: practodev/workerpodautoscaler:v1

Exact version are the best but a lot of people use it like this also.

Docker for Mac: standard_init_linux.go:228: exec user process caused: exec format error

Hi,

I am unable to run the container built with make container. See Docker version and run output below. The container works fine when I run it manually using ./bin/darwin_amd64/server. How can I solve this?

docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:22 2021
 OS/Arch:           darwin/amd64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker run example.com/server:0.0.10__darwin_amd64
standard_init_linux.go:228: exec user process caused: exec format error

Running make test

Hi Tim, when running the make test target on my macbook I get the following error. Is this something you've come across previously?

$ go version
go version go1.9.1 darwin/amd64

$ docker --version
Docker version 17.09.0-ce, build afdb6d4

$ docker images
REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
thockin/myapp-amd64                                    7de82d3             94b0c74af6b5        19 minutes ago      5.92MB

doneill at doneill-mbp in /projects/github/go-build-template on master
$ make test
/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/bin/sh: -c: line 1: syntax error: unexpected end of file
make: *** [test] Error 2

Invoking the script manually outside of the docker image, from my shell.

$ ./build/test.sh cmd pkg
Running tests:
?   	_/projects/github/go-build-template/cmd/myapp	[no test files]
?   	_/projects/github/go-build-template/pkg/version	[no test files]

-n Checking gofmt:
PASS

-n Checking go vet:
PASS

Can we mark this repository as a template repository?

Ref: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository

go install runtime/internal/sys: mkdir /usr/local/go/pkg/linux_amd64_static/runtime: permission denied

Hello! I'm trying to build a project that uses this template on OS X and I'm getting the following error when I run make container

building: bin/amd64/honeycomb-kubernetes-agent
go install runtime/internal/sys: mkdir /usr/local/go/pkg/linux_amd64_static/runtime: permission denied
make: *** [bin/amd64/honeycomb-kubernetes-agent] Error 1

This article (https://dave.cheney.net/2015/08/22/cross-compilation-with-go-1-5) makes it seem like it's a bad idea to use go install when cross compiling, is that an issue you've seen before? Should I just make /usr/local/go/pkg writable and see what happens? The article also suggests building go from source but it's not clear to me how that would help.

Compatibility with Drone CI

Thanks for this repo, I've been really loving how easy it makes getting projects up off the ground! I'm curious about how this would work in a container based CI environment like Drone.

In Jenkins or Travis, using Make directly seems like a clear course of action. But what about when using something like Drone where the assumption is that your build is going to take place in an ephemeral Docker container already?

I'd like to leave Make in place for local folks that aren't hooked into Drone but I'm not sure the best way to make the two workflows play nice next to each other. My current thought is to try to offload drone's build step directly to the shell scripts in ./build and filling in the environment variables where needed, but maybe I'm missing a better option? Maybe it's just the case that using a skeleton like this remove the utility of a service like Drone or on the flipside, using something like Drone means having to go all in, including locally.

Curious to hear folks' thoughts.

Consider: figure out generated files (at least protoc)

It will be more and more common to have generated files and things like protoc. Protoc itself is super complicated to get working correctly and really lends itself to containerized building.

Other generated files (go generate) aren't quite so complicated and it should be easy to explicitly run those.

Consider: don't use host bind mounts at all

Using host bind mounts only works well when you are on Linux. Docker for Mac is super slow. It also means that your local workstation has to be your local build machine.

We should explore how much more complicated it is to not user local bind mounts and explicitly transfer input/output to container.

The experience with Kubernetes suggests it isn't worth it for smaller projects but we should look regardless.

Related: http://docker-sync.io/

SRC_DIRS

The README.md says

maybe change SRC_DIRS if you use some other layout

But SRC_DIRS does not exist in the Makefile or anywhere in the repo.

Should this be removed from the README?

Switch to multiarch/alpine for baseimages

It seems that Alpine now supports other platforms.

Distribute binaries with a one liner script

Script to install the binary with a one liner bash command.

Goals

• Script should work in all platforms - macOS, linux..
• Script should work both in private Github repos and public repos.

I was writing a golang project and i had to distribute binaries - so have written some scripts to do it. It is a very common use case and it would be nice to have it here. :)

Something like (below works in non private repos - to make them work in private repos you need to use github assets api)

• https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-with-curl-on-linux
• https://github.com/vapor-ware/ksync/#linuxmacosx

make build binary name

I changed the variable BIN to

# The binary to build (just the basename).
BIN := planner

When doing a build

$ make build
building: bin/amd64/planner

Results in binary named myapp, should this not be a binary called planner (in my case)?

$ tree bin
bin
└── amd64
    ├── darwin_amd64
    └── myapp

2 directories, 1 file

Feature: Run a bash shell inside build container

Sometimes you want to run arbitrary go commands in the build container. Having a way to drop in to a shell there is super useful.

This could be useful for dealing with the go generate case. See #2.

Add some configurable flag for using CGO (while keep building a static binary)

The overall goal is to enable to use of boringcrypto. While this requires some other changes, they should be possible with Make flag overrides, except for this CGO one.

Using multiarch manifest as BASEIMAGE doesn't just work

It's attractive to put multiarch manifest path in the BASEIMAGE variable, especially when some images (eg. distroless-iptables) don't have arch-suffixed versions. However this use doesn't just work, since docker build runs on the native arch and always fetch the native base image and build for all platforms on top of it.

Work flow - go get

I was wondering how you manage dependant packages when compiling within the docker container?

golangci-lint issues

Issue1

go 1.18 support is not there in linter yet golangci/golangci-lint#2649, so these start failing https://github.com/thockin/go-build-template/blob/master/.github/workflows/lint.yaml#L39 if imports are of generic code

Issue2

Github action is using

- name: Lint
        uses: golangci/golangci-lint-action@v2

and the local make lint is using

require github.com/golangci/golangci-lint v1.47.2

It is failing as lint.sh needs the same parameters as github actions to show the same behaviour, fixed in my local by adding golangci.yaml and adding this to lint.sh: ERRS=$(golangci-lint run -v --max-same-issues 10 --disable-all --exclude-use-default=false "$@" 2>&1 || true)

I can push the fix if you are ok with it ^

bash-specific syntax in build.sh (maybe others) with shebang line `#!/bin/sh`

So if the build image is switched to something with sh symlinked to others (eg. dash, in google-go.pkg.dev/golang), build fails.

make all 
# building for linux/amd64
./build/build.sh: 19: set: Illegal option -o pipefail
make: *** [Makefile:157: go-build] Error 2

We should consider removing bash-specific syntax, or declaring the shebang line as #!/bin/bash.