Missing dependencies (e.g. ghodss/yaml missing in kubernetes/apimachinery)

[albert0815]

Missing dependencies (e.g. ghodss/yaml missing in kubernetes/apimachinery)

[nscuro]

Hi @albert0815, great question!

The tl;dr is that everything is working as expected, and here's why:

Go is a lot smarter than other ecosystems when it comes to detecting which dependencies are actually needed. There are lots of situations where a module may be included in the output of cat go.mod, cat go.sum, go list -m all, or go mod graph, despite there not being an actual dependency. When building, Go analyzes all imports transitively. If no package in a module is ever imported in the "active" code, that module is not included in the build.

Go provides the go mod why command, that

[...] shows a shortest path in the import graph from the main module to each of the listed packages

It also has the -m flag, which allows to query for modules rather than for packages:

The -m flag causes go mod why to treat its arguments as a list of modules. go mod why will print a path to any package in each of the modules

If we query k8s.io/apimachinery for k8s.io/kube-openapi, we get:

$ go mod why -m k8s.io/kube-openapi
# k8s.io/kube-openapi
k8s.io/apimachinery/pkg/util/managedfields
k8s.io/kube-openapi/pkg/schemaconv

So k8s.io/kube-openapi is required because the k8s.io/apimachinery/pkg/util/managedfields package depends on k8s.io/kube-openapi/pkg/schemaconv. If we repeat this for github.com/ghodss/yaml, we get:

$ go mod why -m github.com/ghodss/yaml
# github.com/ghodss/yaml
(main module does not need module github.com/ghodss/yaml)

OK, github.com/ghodss/yaml is not required at all by k8s.io/apimachinery. As a sanity check, we can repeat this for gopkg.in/yaml.v2:

$ go mod why -m gopkg.in/yaml.v2
# gopkg.in/yaml.v2
k8s.io/apimachinery/pkg/runtime
sigs.k8s.io/structured-merge-diff/v4/value
gopkg.in/yaml.v2

We now see that transitive relationships are indeed discovered, but for some reason github.com/ghodss/yaml is not recognized. Why?

Well, switching to k8s.io/kube-openapi now, we repeat the procedure above:

$ go mod why -m github.com/ghodss/yaml
# github.com/ghodss/yaml
k8s.io/kube-openapi/test/integration/builder3
github.com/getkin/kin-openapi/openapi3
github.com/ghodss/yaml

This tells us that github.com/ghodss/yaml is required by github.com/getkin/kin-openapi/openapi3, which in turn is imported by k8s.io/kube-openapi/test/integration/builder3. builder3 is an application (package main) and is apparently used to generate test data for kube-openapi's integration tests. The main package can not be imported by other packages, so there is no path in the "import graph" that could lead to k8s.io/apimachinery depending on k8s.io/kube-openapi/test/integration/builder3.

Output of cyclonedx-gomod mod thus does not include github.com/ghodss/yaml, because it uses go mod why -m to prune the dependency graph from cases like this. You can see this process happening if you use the -verbose flag:

10:52AM DBG loading modules includeTest=false moduleDir=.
10:52AM DBG executing command cmd="/opt/homebrew/bin/go list -mod readonly -json -m all" dir=.
10:52AM DBG filtering modules includeTest=false moduleCount=87 moduleDir=.
10:52AM DBG executing command cmd="/opt/homebrew/bin/go mod why -m -vendor [...] github.com/ghodss/yaml" dir=.
10:52AM DBG filtering module module=github.com/ghodss/yaml reason="not needed"

Most other tools will just read the go.mod or go.sum file. Maybe they also run go list -m, but all these strategies lead to unbearable noise. You shouldn't have to worry about github.com/ghodss/yaml if there's no way for your module to ever use it.

Hope this helps, please let me know if you need further clarification.

[albert0815]

Thank you very much for the detailled explanation! I got it, closing the ticket.