readthedocs/common

Shared bits around multiple repositories

Read the Docs common development files

Common shared bits for development and release tooling across multiple repositories.

Installing

To install:

git submodule add [email protected]:rtfd/common.git common
git submodule update
make -f common/common.mk

Setup

The release process automatically handles updating the changelog and incrementing a version in setup.cfg. You'll need a few extra piece to make this work in each repository: a version to manipulate and information on the repo for changelog automation.

There are several options that can come under the tool:release section:

github_owner

The GitHub repository owner name

github_repo

The GitHub repository name

github_private

Is the repository private? This also requires the use of a GITHUB_TOKEN environment variable when pulling changelog. You can generate this at: https://github.com/settings/tokens

Additionally, setup.cfg can track all of the setup.py options in plaintext. For an example of how to migrate all the setup.py options to setup.cfg, see readthedocs.org or readthedocs-corporate.

Here's an example configuration:

[metadata]
version = 1.0.0

[tool:release]
github_owner = rtfd
github_repo = readthedocs.org
github_private = False

common's Issues

Development: remove initial banner on Celery processes

I found there is an option to remove the initial banner in Celery, but I'm not sure how to pass that argument. See celery/celery#3496

build_1        | Please specify a different user using the --uid option.
build_1        | 
build_1        | User information: uid=0 euid=0 gid=0 egid=0
build_1        | 
build_1        |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
build_1        |  
build_1        |  -------------- celery@2bc10ea96dd8 v5.2.7 (dawn-chorus)
build_1        | --- ***** ----- 
build_1        | -- ******* ---- Linux-6.6.18-1-lts-x86_64-with-glibc2.35 2024-03-06 15:33:00
build_1        | - *** --- * --- 
build_1        | - ** ---------- [config]
build_1        | - ** ---------- .> app:         readthedocs:0x709d4266b640
build_1        | - ** ---------- .> transport:   redis://:**@cache:6379/0
build_1        | - ** ---------- .> results:     disabled://
build_1        | - *** --- * --- .> concurrency: 1 (prefork)
build_1        | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
build_1        | --- ***** ----- 
build_1        |  -------------- [queues]
build_1        |                 .> build01          exchange=build01(direct) key=build01
build_1        |                 .> build:default    exchange=build:default(direct) key=build:default
build_1        |                 .> build:large      exchange=build:large(direct) key=build:large
build_1        |                 .> builder          exchange=builder(direct) key=builder
build_1        |                 .> celery           exchange=celery(direct) key=celery
build_1        |                 .> default          exchange=default(direct) key=default
build_1        |

It's a little tedious to see this message over and over again each time you modify a file when working locally.

Replace versioning with bump2version

A neat idea was brought up in a theme PR: readthedocs/sphinx_rtd_theme#822

It would replace code that we have for loading the version from setup.cfg perhaps, but would also replace the invoke code around cutting a release.

Remove common tasks.py

We don't seem to be using the implementations for version updating or changelog generation from the tasks.py in this repository anymore. It seems these files aren't needed anymore and are replaced by the deploy scripts in readthedocs-ops/deploy instead.

@humitos can you verify?

Configuration for public/private storage policies

In #144, we moved the configuration of MinIO storage bucket policies to a process related to initialize the web container. This was a kind "configuration as code" chosen over the manual setup from before.

Some buckets are private, depending on the environment. This should be expressed as configuration somewhere.

#144 (comment)
#144 (comment)

This issue might be solved in a way that only affects the docker setup, and if possible/necessary can be integrated for other deployments as well.

Share .gitignore

I often have to recreate a bunch of random files in the gitignore. Does it make sense for us to share this across repos?

Flip running flag for some/all docker tasks

One thing that confused me about the docker tasks when i was starting was that the tasks usually all spun up a new instance of web/whatever to run taks like manage and shell. I expected the commands to run in an existing container and it took me a long time to hunt down several bugs because these commands were executed in a fresh container.

One issue that the team will hit is that if you are running the instance with inv docker.up --ext-theme, you can't run inv docker.manage collectstatic to collect assets, as the new instance that is spun up doesn't have the theme enabled and the new assets won't be collected.

It seems like running commands in a running container should be the default, not the exception. Running commands in a fresh container is used to run commands like manage commands without an instance running, which seems edge case to me.

What does the rest of the team do here? I always run an instance if I am doing development. It would be safe for me to always flip the running flag to True on all invoke tasks.

Development: share pre-commit configs on frontend projects

We may want to generalize the implementation I've done in addons and share it with other projects. See readthedocs/addons#116 (review)

Issues with current pre-commit

Files are reported with fixed errors, but are never changed. Our theory is the file is changed back and forth a few times in the pass?

Add trailing commas......................................................Failed
hookid: add-trailing-comma

Files were modified by this hook. Additional output:

Rewriting readthedocs/restapi/client.py

autoflake................................................................Passed
yapf.....................................................................Failed
hookid: yapf

Files were modified by this hook.

Sort python imports......................................................Failed
hookid: python-import-sorter

Files were modified by this hook. Additional output:

ERROR: /Users/anthony/secure/dev/readthedocs.org/readthedocs/restapi/client.py Imports are incorrectly sorted.
Fixing /Users/anthony/secure/dev/readthedocs.org/readthedocs/restapi/client.py

docformatter.............................................................Passed
autopep8 wrapper.........................................................Passed
Check for added large files..............................................Passed
Debug Statements (Python)................................................Passed
Fix double quoted strings................................................Passed
Fix End of Files.........................................................Passed
Fix python encoding pragma...............................................Passed
Check for merge conflicts................................................Passed
Check for broken symlinks............................(no files to check)Skipped
Trim Trailing Whitespace.................................................Passed
Flake8...................................................................Failed
hookid: flake8

Traceback (most recent call last):
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2330, in resolve
    return functools.reduce(getattr, self.attrs, module)
AttributeError: module 'pycodestyle' has no attribute 'break_around_binary_operator'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/plugins/manager.py", line 178, in load_plugin
    self._load(verify_requirements)
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/plugins/manager.py", line 150, in _load
    self._plugin = resolve()
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2332, in resolve
    raise ImportError(str(exc))
ImportError: module 'pycodestyle' has no attribute 'break_around_binary_operator'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/bin/flake8", line 11, in <module>
    sys.exit(main())
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/main/cli.py", line 16, in main
    app.run(argv)
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/main/application.py", line 396, in run
    self._run(argv)
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/main/application.py", line 383, in _run
    self.initialize(argv)
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/main/application.py", line 365, in initialize
    self.find_plugins()
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/main/application.py", line 191, in find_plugins
    self.check_plugins.load_plugins()
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/plugins/manager.py", line 423, in load_plugins
    plugins = list(self.manager.map(load_plugin))
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/plugins/manager.py", line 311, in map
    yield func(self.plugins[name], *args, **kwargs)
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/plugins/manager.py", line 421, in load_plugin
    return plugin.load_plugin()
  File "/Users/anthony/.cache/pre-commit/repo3p_jue5e/py_env-python3.6/lib/python3.6/site-packages/flake8/plugins/manager.py", line 186, in load_plugin
    raise failed_to_load
flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "pycodestyle.break_around_binary_operator" due to module 'pycodestyle' has no attribute 'break_around_binary_operator'.

prospector...............................................................Passed

Unhide files

I prefer the pattern of not hiding these files when they are in a subdirectory. It's helpful to not see them in the main repo to avoid polluting that path, but we don't need to worry about that here.

For example, instead of:

- .eslintrc
- .isort.cfg
...

It would be:

- eslintrc
- isort.cfg
...

refs readthedocs/readthedocs.org#3808

Development: reduce the number of files watched by `nodemon`

In theory, nodemon should watch only 779,

root@716db483de08:/usr/src/app/checkouts/readthedocs.org# find ./readthedocs -type f -name '*.py' | wc -l
779

but it says it's watching ~5k:

web_1       | [nodemon] watching 4989 files

Related #160

Add sanity check before release

See discussion here:
#4 (comment)

Raised in #4

Small issues with yapf rules

A couple of small changes that surfaced linting the entire readthedocsinc codebase:

• No need for utf8 header in files anymore
• https://github.com/rtfd/readthedocs-corporate/pull/463/files/5c179ca89339adb3c59d43ab8cc36044f1146708..2f9f47fc69fb2df467f4f17261cf8613850f3d7a#diff-b48d75844df80305a816a632c69f63fd
• https://github.com/rtfd/readthedocs-corporate/pull/463/files/5c179ca89339adb3c59d43ab8cc36044f1146708..2f9f47fc69fb2df467f4f17261cf8613850f3d7a#diff-9bb2189bdb7f183f9a7f6731866b3677

Find a way to share pre-commit/prospector config with .org and .com

Currently our pre-commit settings for prospector doesn't really work well.

Can't be re-used in .com, we had to disable that check https://github.com/readthedocs/readthedocs-corporate/pull/1698. The reason it fails is because of two things:

• The Python path: when running on CI, we clone the readthedocs.org repo in the .tox/py310/readthedocs.org directory, but in dev, we have this repo in the upper directory (../readthedocs.org).
• Requirements: prospector creates a separate environment to run in, due to this, we need to install the requirements inside that environment, and since we are sharing this file for both of our repos, we ended up installing the requirements from a URL, this is a problem if a new dep is added in a PR, or if .com needs an additional dep that isn't on .org.
  

  common/pre-commit-config.yaml

  Line 101 in e598102

  - https://raw.githubusercontent.com/readthedocs/readthedocs.org/main/requirements/testing.txt

Remove volume mounted on builds

We shouldn't require this anymore now that we are running the git commands inside the Docker container.

This mount could probably removed completely: https://github.com/readthedocs/common/blob/master/dockerfiles/docker-compose.yml#L110

Allows to avoid VersionLocked exception when working locally with multiple builders (--scale-build 5).

wrangler: ability to serve addons from local development

The idea would be to call inv docker.up --addons-local-dev or similar and make the CF worker to inject http://localhost:8000/readthedocs-addons.js instead of the /_/static/ URL.

See readthedocs/readthedocs.org#10965 (comment) for more context.

Update release task

• Verify version == version in setup.cfg
• Add git commit step (git commit -m "Release {version}")
• Add git push master
• Do tagging as normal

Fix Docker Compose v2 issues

Details

I haven't been able to make our docker-compose specification work with Docker Compose v2.

It seems there might be an issue with the way that the Docker Compose builder builds and labels images or how it loads those images later.

Problem commandline outline:

COMPOSE_PROJECT_NAME=community inv docker.up --init
[+] Running 0/22
 ⠿ build Error                                                                                                                          1.6s
 ⠿ web Error                                                                                                                            1.6s
 ⠿ celery Error                                                                                                                         1.6s
 ⠧ cache Pulling                                                                                                                        1.6s
   ⠸ 8559a31e96f4 Waiting                                                                                                               0.2s
   ⠸ 85a6a5c53ff0 Waiting                                                                                                               0.2s
   ⠸ b69876b7abed Waiting                                                                                                               0.2s
   ⠸ a72d84b9df6a Waiting                                                                                                               0.2s
   ⠸ 5ce7b314b19c Waiting                                                                                                               0.2s
   ⠸ 04c4bfb0b023 Pulling fs layer                                                                                                      0.2s
 ⠿ proxito Error                                                                                                                        1.6s
 ⠧ database Pulling                                                                                                                     1.6s
 ⠧ nginx Pulling                                                                                                                        1.6s
 ⠿ search Error                                                                                                                         1.6s
 ⠦ storage Pulling                                                                                                                      1.6s
   ⠸ fdf7c3c4a3fe Pulling fs layer                                                                                                      0.3s
   ⠸ 4af7314918ee Pulling fs layer                                                                                                      0.3s
   ⠸ 992a193f98f7 Pulling fs layer                                                                                                      0.3s
   ⠸ 95cf6d3796df Waiting                                                                                                               0.3s
   ⠸ 38d15d2268e2 Waiting                                                                                                               0.3s
   ⠸ f2eb043bf4aa Waiting                                                                                                               0.3s
   ⠸ f46ceb6e5389 Waiting                                                                                                               0.3s
Error response from daemon: pull access denied for community_server, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

There's a series of Docker Compose issues open that seem related but they all have workarounds. I'm expanding this issue once I diagnose this further.