matomo-org/github-action-tests

tests package for github action

Matomo GitHub Action Tests

This action is able to run certain test suites for Matomo or any Matomo plugin.

Inputs

• test-type

  Specifies the test types to run. This can be any of the following:

  • UI
  • PluginTests
  • JS
  • Client
  • UnitTests
  • SystemTestsPlugins
  • SystemTestsCore
  • IntegrationTestsCore
  • IntegrationTestsPlugins

• plugin-name

  Needs to be provided when running tests for a certain plugin only. If not provided tests will run for Matomo itself.

• matomo-test-branch

  When running tests for a plugin, this option defines which version of Matomo should be used for running the tests.

  It can either be a specific branch or tag name (like 4.x-dev or 4.13.1) or one of this magic keywords:

  • minimum_required_matomo

    This will automatically try to determine the minimum required Matomo version for your plugin. This is done by looking at the version requirement in plugin.json

    If a version is defined in plugin.json this version will be tried to check out. If e.g. >=4.0.0-b1,<5.0.0-b1 is defined it will try to check out 4.0.0-b1.

    In case the defined version can not be found. e.g. the tag 4.0.0-b1 is not available, it will first try to check out the stable version (if beta provided). e.g. 4.0.0 in this example. If that would also fail it falls back to the development branch of that major version. So 4.x-dev in that case.

  • maximum_supported_matomo

    This will automatically try to identify the maximum supported Matomo version for your plugin. This is also done by looking at the version requirement in plugin.json

    If a specific version is defined in plugin.json this version will be tried to check out. If e.g. >=4.0.0-b1,<4.7.0 is defined it will try to check out 4.7.0 if a newer version has already been released. If no newer version is available it falls back using the development branch like below.

    In case the upper bound defines that the plugin is compatible with a full major version, e.g. >=4.0.0-b1,<5.0.0-b1, tests will automatically run against the development branch of the supported major version. In this case it would be 4.x-dev.

    Should the defined limits of a plugin contain more than one major releases, e.g. >=4.4.0,<6.0.0-b1, the development branch of the latest support Matomo version will be used. 5.x-dev in that case.

• php-version

  Defines the PHP version to set up for testing. (Not needed for Client tests)

  The action uses shivammathur/setup-php to set up PHP. You can find supported PHP versions here: https://github.com/shivammathur/setup-php#tada-php-support

• node-version

  Defines the Node version to set up for testing. (Not needed for PHP tests)

• redis-service

  Defines if a redis master and sentinel server should be set up before testing.

• mysql-service

  Defines if a MySQL server should be set up before testing. If so a MySQL 5.7 server using tmpfs will be set up.

• mysql-driver

  Defines which Mysql adapter Matomo should use to connect to the database. Can be set to PDO_MYSQL (default) or MYSQLI.

• upload-artifacts

  If set to true produced artifacts will be uploaded to https://build-artifacts.matomo.org

  This is only relevant for test types: UI, PluginTests, SystemTestsPlugins and SystemTestsCore

• artifacts-protected

  If artifacts should be uploaded to https://build-artifacts.matomo.org this option defines if the artifacts should be hidden behind a login. Only needed for premium plugins.

• artifacts-pass

  If artifacts should be uploaded to https://build-artifacts.matomo.org this option needs to contain the correct upload password.

• dependent-plugins

  Additional plugins to check out before testing. Plugins need to be provided as a comma separated list of their slugs.

  E.g. "matomo-org/plugin-CustomVariables" or "matomo-org/plugin-CustomVariables,nickname/PluginName"

  Repositories must be named as PluginName or plugin-PluginName for this to work.

• github-token

  If a dependant plugin is a private repository, this option needs to contain a GitHub access token having access to that repo.

  For security reasons this option should not be provided in plain text, but using a repository secret instead.

• setup-script

  This option can contain the path to a bash script that should be executed before running the tests.

  This can be used by plugin to set up additional requirements. Like e.g. LoginLdap plugin requires a Ldap server for running tests.

• ui-test-options

  Additional options to provide for UI tests. This can be used to split up UI tests in multiple builds like used in core.

• phpunit-test-options

  Additional options to provide for PHPUnit tests. This can be used to provide debugging options for PHPUnit.

Example usage for a Plugin

  PluginTests:
    runs-on: ubuntu-20.04
    strategy:
      fail-fast: false
      matrix:
        php: [ '7.2', '8.1' ]
        target: ['minimum_required_matomo', 'maximum_supported_matomo']
    steps:
      - uses: actions/checkout@v3
        with:
          lfs: true
          persist-credentials: false
      - name: Run tests
        uses: matomo-org/github-action-tests@main
        with:
          plugin-name: 'PluginName'
          php-version: ${{ matrix.php }}
          test-type: 'PluginTests'
          matomo-test-branch: ${{ matrix.target }}
          artifacts-pass: ${{ secrets.ARTIFACTS_PASS }}
          upload-artifacts: ${{ matrix.php == '7.2' && matrix.target == 'maximum_supported_matomo' }}
          artifacts-protected: true
          dependent-plugins: 'slug/plugin-AdditionalPlugin'
          github-token: ${{ secrets.TESTS_ACCESS_TOKEN || secrets.GITHUB_TOKEN }}

github-action-tests's Issues

Plugin tests do not run if group tag is missing

I just noticed 10+ test classes weren't running on Travis in one of our plugins because they were missing a header like below above the class

/**
 * @group MyPluginName
 * @group Plugins
 */
class myclasstest

When the @group $PluginName is not there, then it seems the tests aren't executed on Travis. We actually had some failing tests that gone unnoticed.

Can we change this that all tests in a plugin directory are considered to be of that group and executed by default?

I assume the same issue will be when there's github action tests instead of travis.

cc @mattab might be something to prioritise.

A child feature branch merging into its parent feature branch breaks plugin version checkout

I'll leave this one here so we have it documented as a possible problem.

When a child feature branch is merging into its parent feature branch will cause the dependent plugin to checkout the latest version. In my case it causes 5.x to be checked out when its parent is based off 4.x. This causes some troubles with testing of course.

Recreation

• Create a feature branch feature-a based off a 4.x-dev plugin branch.
• Create another feature branch feature-b based off feature-a
• Run the tests for feature-b

Actual: 5.x is checked out for the dependent plugins
Expected: 4.x version of the dependent plugin should be checked out.

Perhaps we can allow the base version to be defined so that this line will honor the intended Matomo version.

github-action-tests/action.yml

Line 185 in d52ff9f

TARGET_BRANCH: ${{ github.base_ref || github.ref_name }}

plugin builds do not copy hidden files to /path/to/matomo/plugins/$PLUGIN_NAME

Plugin files like matomo/plugins/MyPlugin/.hidden do not get copied into the cloned Matomo in https://github.com/matomo-org/github-action-tests/blob/main/scripts/bash/checkout_matomo.sh#L15.

They should be copied, as they might be or may become necessary for automated tests.

(In my case, the file is named .wp-env.json and is needed for a new test.)

pass token to actions/checkout for private submodules

Currently, if a plugin has a submodule that is a private repository, the generated github action will fail when trying to checkout the repository. This is due to the action trying to init/update submodules without a valid token.

This could be fixed by supplying a token: parameter to actions/checkout like so:

token: ${{ secrets.TESTS_ACCESS_TOKEN || secrets.GITHUB_TOKEN }}

Link to CI build artefacts doesn't work when source and target branch is the same

It's not very common but can happen when community members don't use a new branch and open a PR from their fork within the same branch name.

Example PR — matomo-org/matomo#20969

Target branch: 5.x-dev
Source branch: 5.x-dev

The link to the assets from UI tests is missing the branch as it probably evaluated there's no branch, and therefore the link doesn't present anything (goes to 404).

Stretch goal: When looking into this, we could also address branches with slashes in them to work as expected, which might be a configuration within the builds webserver. Alternatively we can replace slashes with some filesystem-indifferent character.