ferretdb/github-actions

Shared GitHub Actions for FerretDB repos

FerretDB

FerretDB was founded to become the de-facto open-source substitute to MongoDB. FerretDB is an open-source proxy, converting the MongoDB 5.0+ wire protocol queries to SQL - using PostgreSQL or SQLite as a database engine.

flowchart LR
  A["Any application\nAny MongoDB driver"]
  F{{FerretDB}}
  P[(PostgreSQL)]
  S[("SQLite")]

  A -- "MongoDB protocol\nBSON" --> F
  F -- "PostgreSQL protocol\nSQL" --> P
  F -. "SQLite library\nSQL" .-> S

Why do we need FerretDB?

MongoDB was originally an eye-opening technology for many of us developers, empowering us to build applications faster than using relational databases. In its early days, its ease-to-use and well-documented drivers made MongoDB one of the simplest database solutions available. However, as time passed, MongoDB abandoned its open-source roots; changing the license to SSPL - making it unusable for many open source and early-stage commercial projects.

Most MongoDB users do not require any advanced features offered by MongoDB; however, they need an easy-to-use open-source document database solution. Recognizing this, FerretDB is here to fill that gap.

Scope and current state

FerretDB is compatible with MongoDB drivers and popular MongoDB tools. It functions as a drop-in replacement for MongoDB 5.0+ in many cases. Features are constantly being added to further increase compatibility and performance.

We welcome all contributors. See our public roadmap, a list of known differences with MongoDB, and contributing guidelines.

Quickstart

Run this command to start FerretDB with PostgreSQL backend:

docker run -d --rm --name ferretdb -p 27017:27017 ghcr.io/ferretdb/all-in-one

Alternatively, run this command to start FerretDB with SQLite backend:

docker run -d --rm --name ferretdb -p 27017:27017 -e FERRETDB_HANDLER=sqlite ghcr.io/ferretdb/all-in-one

This command will start a container with FerretDB, PostgreSQL/SQLite, and MongoDB Shell for quick testing and experiments. However, it is unsuitable for production use cases because it keeps all data inside and loses it on shutdown. See our Docker quickstart guide for instructions that don't have those problems.

With that container running, you can:

• Connect to it with any MongoDB client application using MongoDB URI mongodb://127.0.0.1:27017/.
• Connect to it using MongoDB Shell by just running mongosh. If you don't have it installed locally, you can run docker exec -it ferretdb mongosh.
• For the PostgreSQL backend, connect to it by running docker exec -it ferretdb psql -U username ferretdb. FerretDB uses PostgreSQL schemas for MongoDB databases. So, if you created some collections in the test database using any MongoDB client, you can switch to it by running SET search_path = 'test'; query and see a list of PostgreSQL tables by running \d psql command.
• For the SQLite backend, connect to it by running docker exec -it ferretdb sqlite3 /state/<database>.sqlite. So, if you created some collections in the test database using any MongoDB client, run docker exec -it ferretdb sqlite3 /state/test.sqlite and see a list of SQLite tables by running .tables command.

You can stop the container with docker stop ferretdb.

We also provide binaries and packages for various Linux distributions, as well as Go library package that embeds FerretDB into your application. See our documentation for more details.

Building and packaging

Note

We strongly advise users not to build FerretDB themselves. Instead, use binaries, Docker images, or packages provided by us.

FerretDB could be built as any other Go program, but a few generated files and build tags could affect it. See there for more details.

Managed FerretDB at cloud providers

• Civo
• Scaleway
• Tembo
• Vultr

Documentation

• Documentation for users.
• Documentation for Go developers about embeddable FerretDB.

Community

• Website and blog: https://www.ferretdb.com/.
• Twitter: @ferret_db.
• Mastodon: @[email protected].
• Slack chat for quick questions.
• GitHub Discussions for longer topics.
• GitHub Issues for bugs and missing features.
• Open Office Hours meeting every Monday at 17:00 UTC at Google Meet.

If you want to contact FerretDB Inc., please use this form.

github-actions's Issues

Check that the PR title starts with an imperative verb

github-actions/conform-pr/main.go

Lines 243 to 251 in 6484839

	// checkTitle checks if PR's title does not end with dot.
	func checkTitle(_ *githubactions.Action, title string) error {
	titleRegexp := regexp.MustCompile("[a-zA-Z0-9`'\"]$")
	if match := titleRegexp.MatchString(title); !match {
	return fmt.Errorf("PR title must end with a latin letter or digit.")
	}
	return nil
	}

That function should also check that title start with an imperative verb ("Fix …", not "Fixing …", "Fixed …" or other forms).

We could probably use https://github.com/jdkato/prose/tree/v2 (despite it being archived) and check that the first token has the correct POS tag.

Check workflow type and token permissions

We should decide between pull_request and pull_request_target workflow type, and between using automatic short-lived GITHUB_TOKEN and PAT.

`conform-pr` should allow one single newline at the end of the PR description

What the title says.

We probably should also replace \r\n with \n before checking it.

Add support for `no ci` label

https://github.com/FerretDB/FerretDB/blob/3a05467dacb0fe01fbf18ff46ddc596f5c5b7de8/.github/workflows/docker.yml#L59

We should have a similar condition in all our workflows that would skip CI completely if no ci label is present.

Enable Go deps caching with setup-go

https://github.com/actions/setup-go#caching-dependency-files-and-build-outputs

Do not add `WEEK_NUMBER` to environment

WEEK_NUMBER is added to all subsequent bash commands, including ones in the caller workflow. That might cause problems. Let's use outputs.

Port our standard tools from FerretDB and dance

• Add and use Taskfile with basic commands
• Use tools module to install tools
• Add very basic golangci-lint configuration

Add contributing guidelines

We have some good contributing guidelines in the FerretDB repo. We should copy relevant parts there and mention github-actions-specific projects.

Read organization members

Some Conform PR checks will be different between PRs from the community and FerretDB organization members. We should read organization members and check if the PR author belongs to it.

Set Sprint field in PRs

Investigate GitHub's GraphQL API

For #45, we will need to start interacting with GitHub Projects (beta) via GraphQL API: https://docs.github.com/en/issues/trying-out-the-new-projects-experience/using-the-api-to-manage-projects

First, let's try querying that API manually via gh tool.

Second, let's investigate what Go modules we should use for that API (probably https://github.com/shurcooL/githubv4?)

Lastly, let's add some code to conform-pr GitHub Action that queries project's iteration for PRs.

Extract multiple Docker tags

For FerretDB/FerretDB#70, we want extract-docker-tag action to be able to return several tags. For PR branches, we should return a single tag like now, but for pushed git tags, we should return two Docker tags: version without v prefix and latest. To simplify migration, that probably should be a separate output field. It will be used by that code in FerretDB repo:

• https://github.com/FerretDB/FerretDB/blob/d48b4fa526675440c534d56bea291a0a652bd4c5/.github/workflows/docker.yml#L97-L98
• https://github.com/FerretDB/FerretDB/blob/d48b4fa526675440c534d56bea291a0a652bd4c5/Taskfile.yml#L421-L449

Add a unit test or test it with your own fork.

Markdown summary is not immediately visible

PR fails:

After one click:

After second click:

I think we could log summary table items to the log to make results more immediate. Bonus points for using tabwriter to make an accurate table.

`extract-docker-tag` does not work with Dependabot's PRs

Dependabot creates branches with names like dependabot/submodules/tests/mongo-go-driver-29d768e that generate invalid tag names. For example: ghcr.io/ferretdb/ferretdb-dev:pr-dependabot/submodules/tests/mongo-go-driver-29d768e (https://github.com/FerretDB/dance/runs/5411721756)

`conform-pr` Action should handle more punctuation

Currently, it checks that the PR title does not end with a dot, but accepts !, ?, whitespace, etc.

Instead, it should check that title matches [a-zA-Z0-9]$ regular expression.

Use organization webhooks and log them

We want our Conform PR GitHub Action to check PR's Project fields after they are updated, but there is no such event to trigger an Action run. That's because Projects v2 are organization-scoped, and Actions are repository-scoped.

The solution is to install an organization webhook to receive Project v2 events, find matching PR, and schedule a new Conform PR run.

Let's create a simple service for it. It should use packages we already use (github.com/google/go-github and github.com/shurcooL/githubv4); it should not use any web or API framework. It should receive incoming webhooks, validate them by checking the signature, check secrets, find matching PR and schedule a new Conform PR run and just log them for now.

Webhook example:

X-Github-Event: projects_v2_item
X-Hub-Signature: sha1=5d9d657f2244700cab8e7996af2ce3ef476d5016
X-Hub-Signature-256: sha256=9c23e56fdae920dc6eb3754cd4f4272e1909b24da6d4ec3440684a6906d28051

{
    "action": "edited",
    "projects_v2_item": {
        "id": 8420853,
        "node_id": "PVTI_lADOBay1Kc4AEwi9zgCAffU",
        "project_node_id": "PVT_kwDOBay1Kc4AEwi9",
        "content_node_id": "PR_kwDOGfwnTc48u60R",
        "content_type": "PullRequest",
        "creator": {
            "login": "AlekSi",
            "id": 11512,
            "node_id": "MDQ6VXNlcjExNTEy"
        },
        "created_at": "2022-08-09T20:36:46Z",
        "updated_at": "2022-08-19T17:46:58Z",
        "archived_at": null
    },
    "changes": {
        "field_value": {
            "field_node_id": "PVTSSF_lADOBay1Kc4AEwi9zgCzrKM",
            "field_type": "single_select"
        }
    },
    "organization": {
        "login": "FerretDB",
        "id": 95204649,
        "node_id": "O_kgDOBay1KQ"
    },
    "sender": {
        "login": "AlekSi",
        "id": 11512,
        "node_id": "MDQ6VXNlcjExNTEy"
    }
}

The PR in question is #85.

`conform-pr`Action should check PR description

It should check that the PR description ends with a dot. In particular, it should not end with whitespace.

`conform-pr` Action should produce a Markdown summary

Something like this:

Check	Status
The title doesn't end with punctuation	✅
The description contains issue link	❌ Issue link should be #23, not full github.com URL

See https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

Make our own git checkout action

That would checkout PR's HEAD SHA instead of merge commit that is not available for pull_request_target.
It also should checkout a named branch.

`conform-pr` fails for titles ending with `

See FerretDB/FerretDB#688. "Handle both buildinfo and buildInfo" should be a valid title.

`extract-docker-tag` should support git tags

Git tag vX.Y.Z should return Docker tag X.Y.Z. Any other tag should produce an error.

conform-pr: check that auto-merge is enabled

conform-pr should fail if PR’s auto-merge is not enabled.

conform-pr: check PR title length

conform-pr action should check that the PR title (that is used as commit title on squash&merge) is not longer than 72 Unicode runes. That would prevent the truncation of commit messages in the GitHub UI. It also will make our changelogs better.

Make an action to extract URL from input file

We have a workflow that builds PR with documentation and deploys it to CloudFlare pages for preview. The way to see that preview URL is currently quite cumbersome. We should improve it.

Write a composite action (similar to extract-docker-tag, but much simpler) that reads "deploy.txt" file, extracts the first encountered URL, and sets the output parameter.

Use that action in the FerretDB workflow to use the deployment URL. See references to this issue.

`detect-matching-pr` does not work correctly in some cases

https://github.com/FerretDB/FerretDB/runs/5410279872

detect: event.Sender.Login "AlekSi" != event.PullRequest.User.Login "seeforschauer"

trust label was added by me, but PR was created by @seeforschauer.

Check Size field in PRs

Conform PR action should check that the pull request does not contain a Project v2 field (across all PR's projects) "Size" with a set value. That's because we agreed that only issues have Sizes, but not PRs.

Rename `ProjectNext` GraphQL objects to `ProjectV2`

https://github.blog/changelog/2022-06-23-the-new-github-issues-june-23rd-update/

That should be as simple as renaming.

Check milestone

conform-pr should check that PR has (any) milestone assigned.

See https://github.com/FerretDB/github-actions/blob/main/conform-pr/.

Add basic PR verification action

Make a simple action in Go that verifies that PR's title does not end with dot as this does not work nicely with automatically generated release notes. Write tests for it.

Why make our own action instead of using some existing one? Because we will extend with our own logic later, which would be better than using (and updating) several external ones.

Check Sprint field in PRs

When conform-pr GitHub Action checks some pull request, it should fail if the Sprint field (in all projects where it exists) is empty or has a value that is not the "current" one.

Check auto-merge commit title

github-actions/conform-pr/main.go

Lines 323 to 335 in 0d831dd

	// checkAutoMerge checks if PR's auto-merge is enabled.
	func checkAutoMerge(_ *githubactions.Action, pr *graphql.PullRequest, community bool) error {
	if pr.Closed || pr.AutoMerge {
	return nil
	}
	msg := `PR should have auto-merge enabled.`
	if community {
	msg += ` Don't worry, maintainers will enable it for you.`
	}
	return errors.New(msg)
	}

conform-pr's auto-merge checker should check that the commit_title property matches PR's subject number plus the PR number with # and brackets. That is to ensure that auto-merge generates the right commit subject line after the PR title is edited.