ahmetb/gen-crd-api-reference-docs

API Reference Docs generator for Kubernetes CRDs (used by Knative, Kubeflow and others)

Kubernetes Custom Resource API Reference Docs generator

If you have a project that is providing Custom Resource Definitions and wanted to generate API Reference Docs like this this tool is for you.

Alternatives

This project has inspired creation of the following projects:

• Kubernetes reference-docs generator: used in official Kubernetes component reference docs

• crd-ref-docs by Elastic: A fresh implementation of this project.

Nowadays, I don't have a lot of time to maintain this tool. So consider using one of the above in case this repo does not work for you.

If you're an open source project, consider exposing your CRD API Reference via https://doc.crds.dev/ without much effort.

Current Users

• Knative API reference docs
• FluxCD API reference docs
• Argo CD operator API reference docs
• Contour ingress controller API reference docs
• Kubeflow API reference docs
• cert-manager API reference docs
• Open Service Mesh API reference docs
• PlanetScale Vitess Operator API reference docs
• Agones API reference docs
• Gardener API reference docs
• New Relic Alert Manager API reference docs
• Antrea API reference docs
• kube-green API reference docs
• Azure Service Operator supported resources
• [[ADD YOUR PROJECT HERE]]

Also some forks:

• elastic/crd-ref-docs: A fresh re-implementation inspired by this project that supports AsciiDoc. Used by Elastic Cloud on Kubernetes API reference docs.

Why

Normally you would want to use the same docs generator as Kubernetes API reference, but here's why I wrote a different parser/generator:

1. Today, Kubernetes API does not provide OpenAPI specs for CRDs (e.g. Knative), therefore the gen-apidocs generator used by Kubernetes won't work.

2. Even when Kubernetes API starts providing OpenAPI specs for CRDs, your CRD must have a validation schema (e.g. Knative API doesn't!)

3. Kubernetes gen-apidocs parser relies on running a kube-apiserver and calling /apis endpoint to get OpenAPI specs to generate docs. This tool doesn't need that!

How

This is a custom API reference docs generator that uses the k8s.io/gengo project to parse types and generate API documentation from it.

Capabilities of this tool include:

• Doesn't depend on OpenAPI specs, or kube-apiserver, or a running cluster.
• Relies only on the Go source code (pkg/apis/**/*.go) to parse API types.
• Can link to other sites for external APIs. For example, if your types have a reference to Kubernetes core/v1.PodSpec, you can link to it.
• Configurable settings to hide certain fields or types entirely from the generated output.
• Either output to a file or start a live http-server (for rapid iteration).
• Supports markdown rendering from godoc type, package and field comments.

Try it out

1. Clone this repository.

2. Make sure you have go1.11+ installed. Then run go build, you should get a gen-crd-api-reference-docs binary executable in the current directory.

3. Clone a Knative repository, set GOPATH correctly, and call the compiled binary within that directory.

   # go into a repository root with GOPATH set. (I use my own script
   # goclone(1) to have a separate GOPATH for each repo I clone.)
   $ goclone knative/build
   
   $ /path/to/gen-crd-api-reference-docs \
       -config "/path/to/example-config.json" \
       -api-dir "github.com/knative/build/pkg/apis/build/v1alpha1" \
       -out-file docs.html

4. Visit docs.html to view the results.

---

This is not an official Google project. See LICENSE.

gen-crd-api-reference-docs's Issues

include a default template with go fs embed

I think we can include the template/ directory in the binary easily now with Go 1.17, and use the custom one if specified via -template-dir or there's a template/ directory in $CWD (not as sure about the latter).

Knative gen-api-reference-docs.sh fail for knative/serving api v0.8

Error:
F0807 13:58:20.621526 168834 main.go:444] type invalid type has kind=Unsupported which is unhandled

Command run:
KNATIVE_EVENTING_COMMIT=v0.8.0 KNATIVE_EVENTING_RESOURCES_COMMIT=v0.8.0 KNATIVE_SERVING_COMMIT=v0.8.0 ./gen-api-reference-docs.sh

Note that Knative/build has been deprecated and removed from our docs and these API builds (thus no Build version in the command).

Comment:
Running the same command but downgrading only knative/serving version to v0.7.0 runs successfully.

FULL LOG:

KNATIVE_EVENTING_COMMIT=v0.8.0 KNATIVE_EVENTING_RESOURCES_COMMIT=v0.8.0 KNATIVE_SERVING_COMMIT=v0.8.0 ./gen-api-reference-docs.sh
go: creating new go.mod: module tmp
go: finding golang.org/x/tools latest
go: finding k8s.io/gengo latest
go: finding golang.org/x/crypto latest
go: finding golang.org/x/net latest
go: finding golang.org/x/sync latest
go: finding golang.org/x/xerrors latest
go: finding golang.org/x/sys latest
Cloning into '/tmp/tmp.dYdA8Mpez8/src/github.com/knative/serving'...
remote: Enumerating objects: 35, done.
remote: Counting objects: 100% (35/35), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 54814 (delta 7), reused 13 (delta 5), pack-reused 54779
Receiving objects: 100% (54814/54814), 37.93 MiB | 14.72 MiB/s, done.
Resolving deltas: 100% (35670/35670), done.
I0807 14:22:34.076743  176988 main.go:123] parsing go packages in directory ./pkg/apis
W0807 14:22:36.279294  176988 parse.go:239] Ignoring child directory github.com/knative/serving/pkg/apis/config/testdata: No files for pkg "github.com/knative/serving/pkg/apis/config/testdata"
I0807 14:22:36.541306  176988 main.go:225] using package=github.com/knative/serving/pkg/apis/autoscaling/v1alpha1
I0807 14:22:36.541341  176988 main.go:225] using package=github.com/knative/serving/pkg/apis/networking/v1alpha1
I0807 14:22:36.541357  176988 main.go:225] using package=github.com/knative/serving/pkg/apis/serving/v1alpha1
I0807 14:22:36.541369  176988 main.go:225] using package=github.com/knative/serving/pkg/apis/serving/v1beta1
F0807 14:22:36.551807  176988 main.go:444] type invalid type has kind=Unsupported which is unhandled
Cleaning up tmp directory: /tmp/tmp.GusHbFc4ox
Cleaning up tmp directory: /tmp/tmp.dYdA8Mpez8

[go1.20] tool doesn't recurse into sub-directories

Unsure why but with go1.20 the tool doesn't recurse into sub-directories anymore.

Script is invoked the following way
https://github.com/knative/eventing/blob/main/hack/update-reference-docs.sh

Output is

I0227 10:51:28.226464   96710 main.go:132] parsing go packages in directory knative.dev/eventing/pkg/apis
F0227 10:51:28.242770   96710 main.go:135] no loaded build package for knative.dev/eventing/pkg/apis
exit status 255

HTML packages produces urlencoded %2f

Hi there. We noticed that the output generated is producing a %2f encoded value instead of a /. That's not a problem for a browser but it can be a problem for any tool looking for validity of references. In our case, it can't validate the page because it thinks it misses the links.

<p>Packages:</p>
<ul>
<li>
<a href="#camel.apache.org%2fv1alpha1">camel.apache.org/v1alpha1</a>
</li>
<li>
<a href="#camel.apache.org%2fv1">camel.apache.org/v1</a>
</li>
</ul>
<h2 id="camel.apache.org/v1alpha1">camel.apache.org/v1alpha1</h2>

An easy workaround we're using is to replace the encoded value, so, the check is not failing anymore.

Is it possible to run directly against OpenAPI v3 schema?

Currently I have directly the OpenAPI v3 schema - is it possible to directly build the docs from that as opposed to the go types?

Emit documentation for typed constants

I posted kubernetes-sigs/gateway-api#173 to generate documentation for service-apis, and I noticed that no documentation is generated for constants of an aliased type.

For example, in the CRD definition, we have:

// GatewayConditionType is a type of condition associated with a Gateway.
type GatewayConditionType string

const (
	// ConditionNoSuchGatewayClass indicates that the specified GatewayClass
	// does not exist.
	ConditionNoSuchGatewayClass GatewayConditionType = "NoSuchGatewayClass"
	// ConditionGatewayNotScheduled indicates that the Gateway has not been
	// scheduled.
	ConditionGatewayNotScheduled GatewayConditionType = "GatewayNotScheduled"
	// ConditionListenersNotReady indicates that at least one of the specified
	// listeners is not ready. If this condition has a status of True, a more
	// detailed ListenerCondition should be present in the corresponding
	// ListenerStatus.
...
)

Documentation is generated for GatewayConditionType, but not for any of the constants that are defined for this type.

<h3 id="networking.x-k8s.io/v1alpha1.GatewayClassConditionType">GatewayClassConditionType
(<code>string</code> alias)</p></h3>
<p>
(<em>Appears on:</em>
<a href="#networking.x-k8s.io/v1alpha1.GatewayClassCondition">GatewayClassCondition</a>)
</p>
<p>
<p>GatewayClassConditionType is the type of status conditions.</p>
</p>
<h3 id="networking.x-k8s.io/v1alpha1.GatewayClassSpec">GatewayClassSpec
</h3>

At least in the way this particular API is written, the doc comments for specific condition types contain information that is useful for API consumers and implementers, so it would be great if these cold be emitted too.

The typeDisplayName (rendered name of Type) is not correct when it is of type Slice of Pointers

One more level of dereferencing is required when the type is slice so as to handle the type like Slice of Pointers better.

Currently, the tool is not creating the correct display type for api fields of type slice of pointers, that is - []*Type. The reference spec is here for the types defined here

The display type of []*GCPDisks should have ideally shown up as either []*GCPDisks (for developer persona) or []GCPDisks (for operator persona) instead of []*github.com/gardener/machine-controller-manager-provider-gcp/pkg/gcp/apis/v1.GCPDisk.

In the main.go here, if the type is a Slice & specifically Slice of Pointers (i.e, Type.Elem.Kind is Pointer), then we have to add another deferencing on t.Elem and return the pointer name.

markdown: md2html code includes closing parens ) in the URL

ref: knative/docs#1552 (comment)

Can not generate projects that uses go module

It used to work with a project that set GOPATH properly. But now if a project has enabled go module, generating doc will fail.

Step to reproduce:

1. git clone https://github.com/rancher/rio
2. refdocs -config ./apidocs/doc-config.json -api-dir "github.com/rancher/rio/pkg/apis/" -out-file ./api-docs.md --template-dir ./apidocs

It will fail

I0821 13:39:55.847585   13796 main.go:123] parsing go packages in directory github.com/rancher/rio/pkg/apis/
W0821 13:39:57.522517   13796 parse.go:224] Ignoring directory github.com/rancher/rio/pkg/apis/: unable to import "github.com/rancher/rio/pkg/apis/": go/build: importGo github.com/rancher/rio/pkg/apis/: exit status 1
go: finding github.com/rancher/rio/pkg/apis latest
go: finding github.com/rancher/rio/pkg latest
can't load package: package github.com/rancher/rio/pkg/apis: unknown import path "github.com/rancher/rio/pkg/apis": cannot find module providing package github.com/rancher/rio/pkg/apis

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x809875]

goroutine 1 [running]:
k8s.io/gengo/parser.(*Builder).AddDirRecursive(0xc0000df590, 0x7ffd46914291, 0x20, 0x6d, 0xc1)
	/home/daishan/go/pkg/mod/k8s.io/[email protected]/parser/parse.go:229 +0xb5
main.parseAPIPackages(0x7ffd46914291, 0x20, 0x91e136, 0x23, 0xc000145f58, 0x1, 0x1)
	/home/daishan/goprojects/gen-crd-api-reference-docs/main.go:197 +0x60
main.main()
	/home/daishan/goprojects/gen-crd-api-reference-docs/main.go:124 +0x2bb

This workflow used to work when we weren't switched to go module.

automatically link type names found in the doc comments

It would be a great enhancement to automatically link any API type names that are found in the doc comments. Adding a markdown link works, but it doesn't render properly in godoc, so if we can add the link in the generator we can have best of both worlds.

Incorrect display of map types

In the generated documentation for Azure Service Operator (ASO), we noticed that the display names of fields with a map type weren't being simplified the same way that other types are.

For example,

On VirtualMachineIdentity_ARM we see a map type with a relative path for the type of map values:

userAssignedIdentities
map[string]./api/compute/v1api20220301.UserAssignedIdentityDetails_ARM

We expected this to display with just the name of the type, given it's in the current package:

userAssignedIdentities
map[string]UserAssignedIdentityDetails_ARM

Slice properties already use this simplified form - several examples can be seen on VirtualMachineInstanceView_STATUS_ARM:

disks
[]DiskInstanceView_STATUS_ARM

extensions
[]VirtualMachineExtensionInstanceView_STATUS_ARM

I've traced the cause of this to the typeDisplayName() function.

Got "Invalid memory address or nil pointer dereference"

I am pretty new to Golang. I've been trying the "Try it out" section in the README for more than one hour and still haven't figured out how to use it successfully.

I have tried different working directories and changed different things. I would describe the closest I can get even though I don't think what I am doing makes sense:

pwd shows I am inside folder_a/gen-crd-api-reference-docs.
Folder structure currently looks like:

+-- folder_a
|     +-- my_project
|            +-- api
|                  +-- component.html
|                  +-- component.md
|            +--pkg
|                 +-- apis
|                        +-- component
|                               +-- v1alpha1
|                                      +-- types.go
|     +-- gen-crd-api-reference-docs
|            +-- gen-crd-api-reference-docs
|            +-- example-config.json

Command I run:

./gen-crd-api-reference-docs \
-config "./example-config.json" \
-api-dir "github.com/knative/build/pkg/apis/build/v1alpha1" \
-out-file docs.html

my GOPATH: "/Users/myname/go"
Error: invalid memory address or nil pointer dereference

I am currently just learning how to use it, but my end goal is to make changes to component.md and component.html based on types.go

What did I do wrong? How can I fix it? Thanks for your help in advance!

Add flag to sort APIGroups in output

This way the resulted file would be the same each CI run for instance.

The issue with more details could be found here: googleforgames/agones#1690

The workaround for my use case would be to sort the output before making decisions was something updated or not.

Relase for latest version

Hi Team,

could we also get a release with all the latest changes, since the last release is for v0.15.0 , it does not cover a lot of things, and I would really like to use the binary directly, rather than creating it inside the docker environment, since that will also force me to add additional packages to the docker image.

Thank you!

leading spaces removed from ``` sections

Hi! The k8s gateway-api project uses this tool to generate its reference docs. Thank you!

One small problem is that yaml samples (which are included in ``` sections) have their leading spaces stripped somewhere in the process so the output is no longer proper yaml.

Here's the input: https://github.com/kubernetes-sigs/gateway-api/blob/e89bbb7042311703d9e076435ef75f15cf8665c2/apis/v1alpha2/httproute_types.go#L120

Here's the output: https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io%2fv1alpha2.HTTPRouteRule

Is there a way to tell gen-crd-api-reference-docs to leave the formatting alone in ``` sections?

Thank you!

Option to generate AsciiDoc output

For Apache Camel-K I've modified this project, adding an option to generate AsciiDoc output rather than html. The current results can be seen here:

https://camel.apache.org/camel-k/next/apis/camel-k.html
https://camel.apache.org/camel-k/next/apis/kamelets.html

The code modifications are:

• a config flag to indicate AsciiDoc output
• some escaping of comment content to avoid interference with Asciidoc table syntax, attribute syntax, and highlighting syntax
• more AsciiDoc-friendly ID names (this is currently always-on, not under control of the flag)
• a method to escape link text, that can be called from templates
• use of text/Template rather than html/Template, controlled by the flag. There is probably a better way than I found to implement this, but as a Go beginner I wasn't able to find it. Advice here would be greatly appreciated.

I also have two sets of templates:

• one is pretty much a translation of the html template to AsciiDoc
• one (used for camel-k) is adapted for use in Antora, with one package per page.

If there's interest in bringing these capabilities into the project, I'd like to start discussing what would be needed to do so. In particular...

• What testing against existing uses would be appropriate?
• Where could one or both sets of AsciiDoc templates go?

Note that #11 includes a separate function that can be called from a template that also produces more AsciiDoc-friendly IDs.

anchorid should not contains with slash

gen-crd-api-reference-docs/main.go

Line 583 in 52e137b

// TODO(ahmetb): currently this is the same as packageDisplayName

packageAnchorID with flash would not work in markdown format

De-duplicate the apiGroup names (or qualify apiVersions) when the type comes from multiple packages

This recently happened in Knative, some types in the same apiGroup come from different Go packages, causing apiGroup listing to be duplicated.

OR maybe I misundersood this, and the types are coming from different apiVersions (like v1alpha1 + v1beta1) which is causing this.

knative/docs#1552 (comment)

Add a --version flag to the binary

Just went through a case where I was debugging an issue while updating this tool.

Was hard to work out which binary version I had on what machine, so would be handy to have a version flag to help with this.

could not get apiVersion for package k8s.io/apiextensions-apiserver/...

I get this error while building my CRD docs.

F0619 10:14:33.489771 188693 main.go:140] could not get apiVersion for package k8s.io/apiextensions-apiserver/pkg/apis/apiextensions: cannot infer kubernetes apiVersion of go package k8s.io/apiextensions-apiserver/pkg/apis/apiextensions (basename "apiextensions" doesn't match expected pattern ^v\d+((alpha|beta)\d+)?$ that's used to determine apiVersion)

Does someone has an idea how to fix or work-around it?

Type aliases get collapsed down into the underlying type.

Over in service-apis, we have a CRD that uses Go type aliases:

type ConfigMapsDefaultLocalObjectReference struct {
...
}

type GatewayClassParametersObjectReference = ConfigMapsDefaultLocalObjectReference
type RouteHostExtensionObjectReference = ConfigMapsDefaultLocalObjectReference
type RouteActionExtensionObjectReference = ConfigMapsDefaultLocalObjectReference

Each alias has unique doc comments describing how it should be used in context.

What happens is that a single documentation block for ConfigMapsDefaultLocalObjectReference gets generated, and it takes the comments from one of the aliases.

https://kubernetes-sigs.github.io/service-apis/spec/#networking.x-k8s.io/v1alpha1.ConfigMapsDefaultLocalObjectReference

What I'm hoping for is for each alias to emit its own unique documentation, and probably inline the fields of the type it is aliasing.

HTML output produced may be not valid

We're experiencing a few problems about HTML validity. Putting below the fragments if they can help you identifying the issue.

• Double p included in the resource descriptions. Validator will complain with No p element in scope but a p end tag seen.

<p>
<p>Package v1alpha1 contains API Schema definitions for the camel v1alpha1 API group</p>
</p>
...
<p>
<p>Kamelet is the Schema for the kamelets API</p>
</p>

• br tag at the end of code. Validator will complain with End tag br.

<code>apiVersion</code></br>
string</td>

• p element nested in h3. Validator will complain with Element p not allowed as child of element h3 in this context.

<h3 id="camel.apache.org/v1alpha1.EndpointType">EndpointType
(<code>string</code> alias)</p></h3>

linkForType and typeDisplayName wrong when a field is a slice of pointers

Given this CRD in Go:

type GitRepositoryStatus struct {
[...]
	// Conditions holds the conditions for the GitRepository.
	// +optional
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// Artifact represents the last successful GitRepository reconciliation.
	// +optional
	Artifact *Artifact `json:"artifact,omitempty"`

	// IncludedArtifacts contains a list of the last successfully included
	// Artifacts as instructed by GitRepositorySpec.Include.
	// +optional
	IncludedArtifacts []*Artifact `json:"includedArtifacts,omitempty"`
}

the output of gen-crd-api-reference-docs is something like

<a href="#source.toolkit.fluxcd.io/v1beta2.*./api/v1beta2.Artifact">[]*./api/v1beta2.Artifact</a>

where the href part is rendered using linkForType .Type and the text is rendered using typeDisplayName .Type.

You can see this happening in the Flux API docs but also in other projects like vitess (search for "[]*").

[wip] Add support for kubebuilder v2

configure rendering of inner table for inner field type

If a field is a type that is linked to a following table, then would it be possible to not render the table for that type within the parent type?
For example:
TFJob.spec -> TFJobSpec, there is a table rendered in TFJob and the same content rendered in a table as TFJobSpec. Seems like the link to the TFJobSpec in TFJob would be enough.