Go modules support?

[google]

Go modules support?

[Bobgy]

go list -m all gives all transitive dependencies with version.

https://blog.golang.org/using-go-modules

[Bobgy]

I built a go license tool for kubeflow.org in
https://github.com/kubeflow/testing/tree/master/py/kubeflow/testing/go-license-tools (at that time, this tool didn't exist).
I don't have much bandwidth to keep maintaining it, so I wanted to discuss if there could be better ways to collaborate.

I took the approach described in this issue:

1. find all dependencies (and transitive dependencies) via go list -m all (for repos using go.mod)
2. infer github repo name from the go import path
3. use github license api to get the license

it doesn't automate as much as this tool did, because

1. inferring github repo url from go import path was hard
2. github license api doesn't recognize some licenses very well, especially BSD 3-clause

but maybe we can improve from there taking sth already built here.

[Bobgy]

Just got a new idea,

1. go mod download -json shows a JSON representation of all the downloaded modules, including their local path (the Dir field)
2. then we can use a tool to scan these dirs for files like LICENSE, license.md, etc like this repo did

go mod download -json example

$ go mod download -json
go: finding github.com/aws/aws-sdk-go v1.14.5
go: finding github.com/aws/aws-lambda-go v1.2.0
{
    "Path": "github.com/aws/aws-lambda-go",
    "Version": "v1.2.0",
    "Info": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.info",
    "GoMod": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.mod",
    "Zip": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.zip",
    "Dir": "/go/pkg/mod/github.com/aws/[email protected]",
    "Sum": "h1:2f0pbAKMNNhvOkjI9BCrwoeIiduSTlYpD0iKEN1neuQ=",
    "GoModSum": "h1:zUsUQhAUjYzR8AuduJPCfhBuKWUaDbQiPOG+ouzmE1A="
}
{
    "Path": "github.com/aws/aws-sdk-go",
    "Version": "v1.14.5",
    "Info": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.info",
    "GoMod": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.mod",
    "Zip": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.zip",
    "Dir": "/go/pkg/mod/github.com/aws/[email protected]",
    "Sum": "h1:+l1m6QH6LypE2kL0p/G0Oh7ceCv+IVQ1h5UEBt2xjjU=",
    "GoModSum": "h1:ZRmQr0FajVIyZ4ZzBYKG5P3ZqPz9IHG41ZoMu1ADI3k="
}

https://stackoverflow.com/a/52082860/8745218

[Bobgy]

Nevermind, I just tried putting a module into GOPATH and run go mod vendor,
after that, running
go-licenses csv <module-name>
generates all its dependencies' licenses.

However, urls are missing as stated

URLs may not be available if the library is not checked out as a Git repository (e.g. as is the case when Go Modules are enabled).

So what I suggested above can already be achieved by this tool.

[Bobgy]

Hmmm, not exactly.

I noticed a dangerous behavior, some repos do not have a license file. e.g. license file is embedded in README: https://github.com/upper/db/tree/v3.0.0.

However, this tool does not give a warning of a module missing license about it when used in this way.

[Bobgy]

I tried to implement ideas in this issue as https://github.com/Bobgy/go-mod-licenses to suit my own use-cases.

[mreiche]

Hey Bobgy, can you explain a little bit more in detail what you achieved?

I just want to license check my local project's dependencies from go.mod, is that the same you were asking for?
When yes, how you would do this?

My approach would be to:

• Read all transitive dependencies
• Perform go license check on every repo
• Collect in CSV and make them unique

[Bobgy]

@mreiche I've got a bit more requirements so did a few more complex things:

• my process requires public URLs of the license files (like this tool provides, but works for go modules), so I had to use go import semantics to figure out corresponding public repos of go modules
• also my process requires scanning every file of each used go module
• also I noticed the full transitive dependency graph of go modules tend to be very large and unnecessary (some dependencies not actually used in built binary), I changed to read the list of modules actually used in a built go binary instead
• there are a bunch more workflow related features to keep configurations for manually verified repos
• and auto redistribute source folders needed
• there're more that I don't have time to explain now

I've implemented a tool like described above and used it to generate licenses, example:

• https://github.com/kubeflow/pipelines/blob/master/v2/third_party/license_info.csv
• https://github.com/kubeflow/pipelines/blob/master/v2/third_party/NOTICES/licenses.txt
• config: https://github.com/kubeflow/pipelines/blob/master/v2/go-mod-licenses.yaml

I want to open source my tool, but I'm currently fairly busy with other stuff. Might not have time to do that soon.

Curious if these sound useful to others.

[Bobgy]

Implemented in #94