Support include/exclude filters

[alecthomas]

Support include/exclude filters

[alecthomas]

I've tried it, it only works with GOPATH :(

[alecthomas]

I forgot to say, this is awesome. I love that it generates static docs, something sorely missing from godoc.

[zigo101]

@alecthomas
Thanks for having interests on Gold!
And thanks for the valuable suggestion / feature request.

Yes, since v0.1.0, the generated docs size becomes much larger, about 2.7x of v0.0.9.
And the number of generated html files becomes 10x of v0.0.9.
This is mainly because now every package-level identifier corresponds a use list page.

As you mentioned, even for v0.0.9-, the number of generated files will be
very large if many 3rd-party packages are depended on. So I think your
request is very reasonable.

However, there are some points I still haven't thought clearly.

1. Is it acceptable to reduce the features when using Gold to generated pages?

In fact, now, the "list unexported types" and "sort by popularity" features are removed in the docs generation mode.
If the "list uses of identifiers" feature (added in v0.1.0) is also removed, then the generated docs size will decrease to 1/3,
and the generated file number will decrease to 1/10.

2. Support JavaScript or not?

To enable the "list unexported types" and "sort by popularity" features,

• without JavaScript, 3x(package count) more pages will be generated.
• with JavaScript, no new pages will be generated (just one more .js file).

3. How to handle the links to the pages for excluded packages?

• option 1: just remove these links. The user experience would be bad.
• option 2: link to the corresponding pages on pkg.go.dev. With this options, there will be still some links needing to be removed, for Gold produces much more links than godoc.
• option 2.a: for std packages, link to docs.go101.org/std, for other packages, still link to pkg.go.dev
• option 3: customized by using a config file, and put the following alike content in it:

  DocsBaseURL[std] = docs.go101.org/std
  DocsBaseURL[A3rdPartyModule] = example.com/bar
  ...
  

Different website might serve html pages generated with different Gold versions, so that there might be some incompatible links.
So option 3 is not very recommended to choose. (For the same reason, option 2.a is also not recommended.)
This problem should be less serious for godoc generated pages, for godoc is comparatively much more stable.

Please note that, in godoc, there are no links from depended package pages to their depending package pages.
But for Gold, this is possible for the relation is two-way. If a depended package is excluded, some information is lost.

Even with the feasible option 2, the use experience will become less happy.
And there are much development (with some unpredictable difficulties) need to be done.

I incline to reduce some features in docs generation mode without excluding any package to get a moderate user experience and to avoid some development difficulties. The docs generated by Gold v0.0.9 for the kubernetes project is 1G without compressing.

I'm listening to your opinions and ideas.

[alecthomas]

Hello!

There are two reasons for my request:

1. Size and time of generation.
2. Confusion for users - they really want the docs for our internal code, not so much for public or generated code. At the moment our internal packages are about 1/100th of the listed packages, so they're a bit hidden in the noise.

BTW during generation, I see the output pausing for a second after every hundred files or so. It looks a lot like a GC pause, or is it something else?

[zigo101]

BTW during generation, I see the output pausing for a second after every hundred files or so. It looks a lot like a GC pause, or is it something else?

Yes, now a manual GC is made for every 1024 page generated. This is mainly to limit the peak memory use in docs generating. For the k8s project, this will make the peak memory use half (from 17G to 8.5G).

However, I found this is helpless in generating std package docs. And this slows down the whole generation about 30%. So, in the next version, the manual GC will be removed. I'm checking why the peak memory use is so high now. If I get the cause, the next version will use a different way to limit the peak memory use.

1. Size and time of generation.

Now, the estimated storage space percentages:

• package docs pages: 1.8 space, 1.7%
• package dep list pages: 0.4% space, 1.7%
• method implementation pages: %1.8 space, 1.7%
• source code pages: 30% space, 4.9% files
• identifier use list pages: 66% space, 90% files

From your description, it looks you don't care about identifier use. So I can add an option to remove this feature in docs generation mode. This will reduce 2/3 storage space, 9/10 file count, and 4/5 generation time.

2. Confusion for users - they really want the docs for our internal code, not so much for public or generated code. At the moment our internal packages are about 1/100th of the listed packages, so they're a bit hidden in the noise.

As I have mentioned in my last comment, it is some hard to remove the docs of dependency packages totally. Doing this will hurt the user experience much. Many users might also need to read the dependency packages to use your pacakges.

How about list your internal packages at the beginning of the index page to emphasize them?

[zigo101]

@alecthomas
I just pushed a new commit, in which several code generation options are added.
You can pull the code and run "go install" to install the dev version.

After the installation, you can run

gold -gen -nouses -plainsrc -emphasize-wdpkgs .

in your project directory to get samll-sized generated docs (about 1/6 of v0.1.0).

Any suggestions and criticizes are still welcome.

[alecthomas]

Brilliant! It complete in ~20s and only takes up 300MB! Thanks :)

[zigo101]

Glad it satisfies you.
I will tag the commit as v0.1.1 if there are no big problems found before tomorrow night.

[zigo101]

I fixed a small problem (images not show) and tag it as v0.1.1.

I will close this issue. However, you can still commit here if any problems are found.

[zigo101]

@alecthomas
FYI, I just found a project which could generate docs by using godoc: https://groups.google.com/g/golang-nuts/c/8IjToAQH5ao

[zigo101]

FYI, the default command name changed to golds (from gold) recently.
And a new option -compact was added to be used as a shortcut of the
combination of several other options, including -nouses and -plainsrc now.
The option might also include other options which will be introduced later.

That means the following command is the same as the aforementioned command.

golds -gen -compact -emphasize-wdpkgs ./...