silverstripe/moduleratings

A check suite package to provide rating metrics for Silverstripe modules

SilverStripe module ratings

This module provides a check suite, and a set of automated checks that can be run over a SilverStripe module to determine a "quality rating".

This library can be installed into a SilverStripe 3 or 4 project that uses Composer.

Installation

You can install this library with Composer if you want to use its public API:

composer require silverstripe/moduleratings

You can also install the silverstripe/moduleratings-plugin Composer plugin, which will provide a Composer command you can run locally to check module quality ratings. Please see the readme in that module for more information on using it.

Requirements

• symfony/finder 3 or 4
• symfony/yaml 3 or 4

Note: If you have conflicts with any of the symfony components (if using the Composer plugin) during installation, you may need to manually require an older version of one or both of those packages, e.g.:

composer require symfony/yaml ~3.0
composer require symfony/console ~3.0

Using the API

To create a check suite, use the SilverStripe\ModuleRatings\CheckSuite class. You will need to provide the directory path to the module code you want to check, and optionally the GitHub repository slug for the module (note: only GitHub supported at this stage). Providing the repository slug will enable checks that look at external continuous integration system APIs to determine quality metrics (e.g. Travis, Scrutinizer).

$checkSuite = new \SilverStripe\ModuleRatings\CheckSuite();

$checkSuite
    ->setModuleRoot('/path/to/silverstripe/framework')
    ->setRepositorySlug('silverstripe/silverstripe-framework');

$checkSuite->run();

echo 'Framework has scored ' . $checkSuite->getScore() . ' out of 100 points. Details:', PHP_EOL;
print_r($checkSuite->getCheckDetails());

The return data from CheckSuite::getCheckDetails is an array with the following example structure (note that example is JSON encoded):

{
  "good_code_coverage": {
    "description": "Has a \"good\" level of code coverage (greater than 40%, requires slug)",
    "points": 5,
    "maximum": 5
  },
  "has_code_of_conduct_file": {
    "description": "Has a code of conduct file",
    "points": 2,
    "maximum": 2
  },
  "coding_standards": {
    "description": "The PHP code in this module passes the SilverStripe lint rules (mostly PSR-2)",
    "points": 0,
    "maximum": 10
  }
 }

Available checks

• "Good" code coverage (>= 40%) (via either Scrutinizer CI or Codecov.io)
• "Great" code coverage (>= 75%) (via either Scrutinizer CI or Codecov.io)
• PHP code is in either "code" or "src" folder
• PHP code passes a PSR-2-ish linting standards check (phpcs.xml.dist is stored in src/Check/CodingStandardCheck for reference)
• Contributing guide file exists
• Has documentation
• EditorConfig configuration file exists
• .gitattributes file exists
• A license file exists
• The repository has a readme
• Scrutinizer CI is configured and has a "good" quality rating (>= 6.5/10)
• One of either Travis CI or CircleCI is configured and the last build passed successfully

The registered checks are defined in config.yml along with the number of points awarded for each check given that it passes.

Caveats

Please note the following caveats/gotchas/todos:

• Code repositories must exist on GitHub for external API checks to work

Thanks!

A huge thank you to Chris Pitt who originally wrote Helpful Robot, the inspiration for this library.

The checks in this library are heavily inspired by the original Helpful Robot checks, and are designed to match the SilverStripe commercially supported module standard.

Without Helpful Robot the SilverStripe community would look a lot less tidy today!

moduleratings's Issues

Check for travis-ci.com

If this is still what's used for rating modules on addons.silverstripe.org (I think it is?) please check travis-ci.com. It hasn't been possible to run builds on travis-ci.org since June 15th 2021.

It's going to show builds as not passing on the addons site even though they are passing on travis-ci.com

Improve support for custom rule-sets

This module, and its companion plugin, could be extended to support custom rule-sets.

A couple of features would allow this:

• Allow the config.yml to be selected, either from one of several bundled with the module, or form a project-specific one
• Allow one config.yml to use another as a base
• Allow checks to have yml-configured properties as well as class names (e.g. for a FilePresenceCheck)

Then the system could be used to build our module style-guide linters, specific to teams / use-cases

Add more checks

For use with #15

composer.json content checks

• Check that a php requirement is defined
• Check that PSR-4 autoloading is configured
• Check that phpunit is installed as a dev dependency
• Check that phpcs is installed as a dev dependency
• Check that type is silverstripe-vendormodule

file presence checks (this could be one, configurable, check class)

• Check for a phpcs.xml.dist
• Check for a phpunit.xml.dist
• Check for a .travis.yml

Update the code quality phpcs standard to use PSR-12 as a base

Currently we use PSR-2 as a base ruleset in src/Check/CodingStandardCheck/phpcs.xml.dist.

We've agreed to push PSR-12 into SS5 and since the changed are also PSR-2 compatible, have been implementing them in SS4 modules as well.

(When) should we switch to using PSR-12 as the base instead of PSR-2?

Move check points into config

Rather than properties of the checks

Travis checks are not reporting correctly

I've noticed a couple of modules that have both the master branch builds passing and the latest build generally having passed which are reporting "Travis passing" as having failed.

This could be an API rate limit from Travis's API

Checking "main" branch, not just "master"

Cause of the new default branch from github it does not count some things like scrutinizer or code coverage.
The simplest solution is to check both branches and take the first where you get a result. For example for scrutinizer i already made a pull request: #24

I will also make another one for code coverage but maybe you have an better idea how to solve the problem (for example a configurable default branch would solve this too)

Update to composer v2

Dependency for silverstripe/addons.silverstripe.org#256

Abstract HTTP requests out so the the API based checks can be mocked

Ideally we can mock HTTP requests to APIs for the Travis, Scrutinizer and Codecov.io based checks.

This would allow us to properly unit test the checks.

Make the logic for file lookups a bit smarter

Instead of looking for variations of a filename in the file based checks, e.g. code_of_conduct.[md/txt/TXT/MD] etc.

We could write a helper method to run various iterations of these if you say provide an input like code of conduct. It could look for code_of_conduct, codeofconduct, code-of-conduct with various extensions (txt, md, no extension) and check across cases.

This wouldn't include checks for .gitattributes and .editorconfig for example since their filenames are specific.

Add CircleCI as a secondary Travis provider if the Travis check returns 404

CircleCI is probably the next most popular CI provider for running tests on SilverStripe modules.

We could rename the TravisCheck to CiBuildsCheck and have it check Travis first then Circle afterwards if Travis returns a 404

Add Scrutinizer CI as a fallback for code coverage checks if Codecov.io is not configured

Similar to #4 we should check for Scrutinizer CI code coverage reports if Codecov.io is a 404, since that's probably more common in SilverStripe 3 modules than Codecov which we're suggesting from SS4 onwards.

Clarification: we'd still recommend using codecov.io on SS3 modules too

User permision

Could it be possible to access a specific group of users?
And each user will vote once per page

Allow the use of guzzle 7

This will make it easier to install in a wider range of projects