cncf/clomonitor

CLOMonitor is a tool that periodically checks open source projects repositories to verify they meet certain project health best practices

clomonitor's Issues

Add link to GitHub page on clomonitor.io

There should be a link to GitHub somewhere on the main https://clomonitor.io page, whether it's in the header as a github octocat icon or "Contribute" pointing to gh/cncf/clomonitor

Add check for "DCO"

Essentially we need to check that the project has this app installed https://github.com/dcoapp/app - however not all projects check for DCO this way or use a CLA only

Another way to do this check is to look at say the last 10 commits and ensure commits as signed off via "git commit -s" - e.g., essentially have the "Signed-off-by: <>" in the git commit signature

Define how scores should be calculated

At the moment we have 4 top level categories (documentation, license, quality and security) as illustrated in the following screenshot:

The global score is calculated based on the average of the 4 categories. The score of each category is defined by the weight of each of the items in the category:

It'd be great to think this carefully and come up with a more balanced formula that reflects the real importance of each item (the current one is just provisional).

Cc: @caniszczyk

Issues with Kubernetes clomonitor page

Looking at https://clomonitor.io/projects/kubernetes/kubernetes

• Artifact Hub badge
  • Not sure what category we would fall into :)
• Developer Certificate of Origin
  • we use CLA instead of DCO so we should not be penalized for this right?
• Slack presence
• fun message here - "Projects should have presence in the CNCF Slack or Kubernetes Slack" ?

Add multiple license scanning tools check (instead of just FOSSA)

CNCF does require license scanning enabled for all projects. However, FOSSA is not the only tool that the project can use, e.g., various projects use Snyk - (cncf/foundation#109 (comment)).

Let's check for the license scanning badge more broadly, not just for the "FOSSA badge".

Add "Public Meeting Minutes" check in "Best Practices"

We expect projects to hold public meetings, e.g., https://goo.gl/5Cergb

A lot of time projects embed this info in the README, there's no standard way we can fetch this information currently.

One idea I have is to stuff it in the landscape.yml on l.cncf.io that we could check for but open for ideas

cc: @amye

Gitpodify clomonitor

In order to make it easier to contribute, would be nice to see if we can gitpodify clomonitor https://www.gitpod.io

Improve Website check

Some projects have their website set at the Org level and not the individual project-level.

This is a valid option and should pass the check.

Example:
https://clomonitor.io/projects/chubao-fs/chubao-fs - Failed website check
https://github.com/cubeFS/cubefs - No site specified here
https://github.com/cubeFS - Site is specified here

Edit: That said, I do want to push projects that have (essentially) a primary repo to also have their website linked. But there are some projects that are relatively large in repo-size, so org-level-linking should be valid.

Provide support to link to artifacts referenced in repositories report

It would be helpful to link to the documents directly in the checklist section.

Check for LF trademark footer for all CNCF project website

This should be in the Legal category

All project sites should have this footer

e.g., https://prometheus.io/

" @ 2022 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page."

For now I think the check can be simple and simply just look for this URL: https://www.linuxfoundation.org/trademark-usage

badges for projects

It would be great if a project can add a clomonitor badge to their project README
e.g., https://clomonitor.io/projects/akri/akri

kind of like the OpenSSF Best Practices badge :)

Enable a way for end users to queue a re-check

The method for this could be up for debate, but initial thoughts:

• Via the site using GH SSO to see if the person is part of the project's org
• A webhook that projects could tie a GitHub action into (and then each project be given an API key of sorts)
• A completely offline check by cloning the clo monitor repo down and running make run locally to queue a fresh pull of everything

  • Ninja edit: Another cooler idea would be allowing them to run some sort of command if they clone down this repo. Think:

    make build
    clomonitor-check kubernetes #established project
    # OR
    clomonitor-check https://github.com/jeefy/tenta # unrelated project
    

    And then output a score via the CLI.

    I think the ideal path would be for someone to check their own score without any action from us.

Release v1.0 of clomonitor

We should have a draft 0.5 release first then v1.0 maybe at kubecon EU?

Show timestamp of latest check, per project

It would be useful to know when a particular project was last checked, especially given that there are different re-check rates for projects -- as I understand it from checks.md:

These checks are run every hour, provided the repository has changed since the last time it was checked. In the case of repositories that don't change often, we make sure that they are checked at least once a day anyway.

/cc @nate-double-u

Add "Website" check in Documentation

We expect our projects to have a public website that's not just a github README

My guess is you can get this data from the github UI which has a field for website

Check for annual reviews

We have a process in CNCF that certain projects need to go through an annual review, we put the output here:
https://github.com/cncf/toc/tree/main/reviews

@amye if you have thoughts on how to make this easier on your end

This is low priority for now until we figure out how to implement

Review linter checks

We need to review if the current linter checks are appropriate and what additional ones could be interesting to add.

Similarly, it'd be good to also review how each of these checks is being done (rules behind each of them) to fine tune them. They can be found hovering over the ? icon on each of them in the project detail view.

Cc: @caniszczyk

Provide support for using governance repos

Provide support for using governance repos instead of only having GOVERNANCE files.

For example; https://github.com/kedacore/governance is where KEDA's governance is at.

Provide support for .github repos

Provide support for .github repos that have the community health resources, such as code of conduct.

Example of KEDA: https://github.com/kedacore/.github/blob/main/CODE_OF_CONDUCT.md

Document how checks work

At the moment we display some of this information next to each check, but it's not exhaustive and not easy go through. We should add a document that explains the different options to pass each of the checks, how often repositories are checked, etc. This should be a handy guide for projects aiming to make CLOMonitor happy.

Related to #151

Kubernetes + SBOM Checks

If I recall, k8s is producing SBOM files somewhere, we should ensure that our check works for it.

@jeefy can look at this

Add "Security" check: project has published a CVE

We should check that a project has published a CVE

https://github.com/containerd/containerd/security/advisories

It shouldn't have a ton of weight but it's a good practice for projects to do so

Support per-repo (or repo type?) configuration

This is somewhat related to #66 but warrants a new Issue/Thread

Many projects have different repos for different purposes. clomonitor should expand the db schema so repos can be flagged as governance, docs, or code repos.

For example, Knative has several repos:

• Community -- A governance repo that holds a lot of policy/procedure info. This should be the primary repo, but it would have a different license (CC 4.0) since it's more of a "Docs" repo. It also is going to fail any sort of "recent release" check since it's not a code repo.
• Eventing -- A code repo that should have an Apache 2.0 License and a "recent release" check
• Service -- A code repo that should have an Apache 2.0 License and a "recent release" check
• Docs -- A docs repo that should have a CC 4.0 License check

If we can decouple the idea of "primary/secondary" repos to instead allow a repo to be flagged as one or more of these categories, I think it will better match many of the projects that exist or will soon be onboarded.

Community bootstrapping

As this has moved forward significantly, we should open this up to the Cloud Native contributor community at large :)

• Schedule a monthly(?) community meeting / office hours (first meeting would be an intro to clomonitor)
• Create #clomonitor Slack channel in CNCF Slack
• Send out comms to maintainers on the purpose of clomonitor, and signal boost the community meeting / slack channel

@tegioz @cynthia-sg I'm happy to tackle all of this :) I just want to make sure you're both aware of it AND make sure when scheduling the meeting it fits in your calendars. Feel free to thread with questions!

/assign

Add CONTRIBUTING.md

Create a contributing.md file that discusses how to contribute to the project, rough architecture + how to contribute linters etc

Increase .clomonitor.yml configuration

Feel free to close this and break it up into separate issues, but I want to capture all the ideas under one since they all involve .clomonitor.yml :)

• We should be able to specify alternative locations for some checks
  For example, https://github.com/project-akri/akri has the header Community, Contributing, and Support, which also links to their Contributing page https://docs.akri.sh/community/contributing.
  Being able to specify in the YAML something like contributing_url: https://docs.akri.sh/community/contributing and pass the check would be great
• We should be able to skip/pass certain checks in .clomonitor.yml. For example(s), Akri or Envoy explicitly state on their docs that they're part of the CNCF. This is technically an accepted path (See cncf/toc#719 (comment))

Add check: Project has at least one release in the last year

Essentially a project should have one release in the last year, e.g.,

https://github.com/artifacthub/hub/releases

We can call it maybe "Recent Release"

Add check for artifact being published on artifacthub

e.g., https://artifacthub.io/packages/helm/prometheus-community/prometheus

We need to find a way to check for this, one idea is to have maintainers put this data in landscape.yml

Check for groups.io presence

(Most) CNCF projects are expected to have a presence on cncf.groups.io. I propose a check to ensure there's at least one mailing list created that follows the pattern of cncf-{project-name}-* (ex. cncf-buildpacks or cncf-buildpacks-dev would both be valid)

There are some exceptions that we need to think about however (mainly, older/graduated projects). Kubernetes uses Google Groups, for example. So perhaps this check should be (initially) only valid on projects that have been adopted starting in 2020 and newer?

Add additional containerd repos

https://github.com/containerd/project -- Governance repo, has their Security/Governance info
https://github.com/containerd/containerd.io -- Their website and docs repo

Edit: Feel free to assign this to me :) Happy to PR it in later.

Detect repo README under `doc` or `.github`

According to the GitHub docs (see About READMEs):

If you put your README file in your repository's root, docs, or hidden .github directory, GitHub will recognize and automatically surface your README to repository visitors.

For an example where the README isn't at the repo's root, see https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation#readme. The clomonitor seems to be only checking the repo root:

/cc @nate-double-u

Enhance security checks with https://github.com/ossf/scorecard

The OpenSSF has the https://github.com/ossf/scorecard project which essentially checks for a bunch of in depth security issues for projects which we could roll up in a score. It appears in places such as https://deps.dev/go/k8s.io%2Fkubernetes

Let's make this low priority for now as there's some overlap with the checks etc that we do.

Produce metrics over time / Reporting ideas

I'm a huge supporter of providing metrics that we can scrape and do reporting on. To that end, I think a basic Prometheus metrics endpoint that publishes the CLO monitor data could be useful.

With that, we could then:

• create reports showing project trajectory over time (And hide projects that have hit some threshold)
• create "alerts" if projects slip or have a negative trajectory
• create "alerts" if a project has not improved their grade over a reasonable amount of time

Theoretically these numbers won't move often, but with every additional project we have more numbers to track.

Add check for 'thelinuxfoundation' user

All CNCF projects require 'thelinuxfoundation' as a user in the project, we should find a way to see if we can check for that

"Quality" change

We should call this "best practices" instead!
Happy to be overruled.

Consider a Makefile for easier development

I'd love it if there were a Makefile so I could simply run make run or make serve off my fork and have a local copy up and running for tinkering. Likewise from there you could have things like make images and make images-push and whatnot tied together.

Add "recheck" button

Kind of a followup to #176. Might it be possible to add a "recheck" button to a project's Clomonitor page? /cc @nate-double-u

Change "Quality" to "Best Practices"

We should change the "Quality" category to "Best Practices"

Manual exceptions

Is there a way to to manually override the checks? For instance at Cilium https://clomonitor.io/projects/cilium/cilium

We have a a USERS.md rather than ADOPTERS.md https://github.com/cilium/cilium/blob/master/USERS.md

and our governance file is in the documentation folder which wasn't part of the scan https://github.com/cilium/cilium/blob/master/Documentation/contributing/governance/commit_access.rst

Stats page for clomonitor

Similar in style to https://artifacthub.io/stats

I'd focus on traffic, projects monitored, repositories etc

Add check for "TRADEMARKS"

CNCF Projects have to xfer their trademark over to the foundation.

There's no easy way to check for this but @amye were chatting that we may have a new policy in CNCF to have a "TRADEMARKS" file at the root of the repo that essentially links to https://www.linuxfoundation.org/trademark-usage/ after we confirm the paperwork is sent our way

Consider supporting checks exemptions

Should we allow repositories to declare some exemptions in certain cases? There are some checks that may not be applicable in some cases (i.e. Artifact Hub badge in the Kubernetes project) and it could be interesting to handle them accordingly (computing scores, displaying a special status in the UI, etc). If we go ahead with this, maybe we should limit the checks this feature would be available for.

Related to #151

Check for Slack Presence

Projects should link to their Slack presence, and said Slack presence is required to be on either the Kubernetes slack (kubernetes.slack.com) or the CNCF Slack (cloud-native.slack.com)

I propose two checks:

1. Slack presence exists at all (by looking through the Repo's markdown files for any Slack links)
2. Slack presence exists on moderated Workspaces (CNCF or K8s)

Edit/Thought: The list of "moderated" workspaces could expand, and so should be able to be updated relatively easily.

Extending Security Checks (and others)

I'd love to see the Security portion of clomonitor include the recommended template files, not as hard and fast checks but as additional visibility of projects exceeding the minimum.
https://contribute.cncf.io/maintainers/github/templates/

For example pass/fail yay/nay for required, and additional items like SECURITY_CONTACTS.md indicated through "dedicated" stars. (2/3 stars for most recommended security templates) (just spitballing here)

Provide support for listing end-users outside of ADOPTERS

Provide support for listing end-users outside of ADOPTERS since KEDA lists everybody on https://keda.sh/community/#users

Support changelogs within GitHub releases

Many projects provide a changelog within the actual GitHub release instead of a markdown file in the main repo. This should be considered a valid path and the tool should support both.

Check for project existence in Project Maintainers list

https://github.com/cncf/foundation/blob/main/project-maintainers.csv houses all the top-level maintainers for all CNCF projects. This is used to do things like allow access to CNCF's Service Desk.

We should validate that a project exists in this list and has at least one maintainer listed.

Review list of approved licenses

At the moment the licenses below (based on the allowed third party licenses document) are considered approved by the license check. It'd be good to review this list and update it as needed.

Apache-2.0
BSD-2-Clause
BSD-2-Clause-FreeBSD
BSD-3-Clause
ISC
MIT
PostgreSQL
Python-2.0
X11
Zlib

Add a way to specific a different location for info

e.g. our roadmap isn't at the root of the repo, it's in "community". It would be nice if there was a way to tell the tool to look some place else. Perhaps via a .clomonitor file? I do think that it would be good to have the tool check to see if the main README included a pointer to the file regardless of where the file lives.

Add check for SBOM output

This is complicated as there are many ways to produce SBOMs, I need to think through how we can check for this but for now wanted to keep this feature request in the backlog