humanmade/altis-dev-tools

Dev Tools module for Altis

altis-dev-tools's Issues

Travis CI builds fail since Composer v2 release

Steps to reproduce:

1. Push a branch
2. See that Travis CI build fails due to Composer v2 being used

The build should pass (providing it passed previously).

Acceptance criteria:

• Travis CI builds pass again

Tidy up our build configuration

When a task is run using TravisCI and navigating to the config tab, there are some minor warnings/info notices witnessed. Just raising a ticket for visibility.

root: deprecated key sudo (The key sudo has no effect anymore.)

root: missing os, using the default linux

root: key matrix is an alias for jobs, using jobs

Use repo import path in .travis.yml instead of direct path

I made a dumb mistake in the .travis.yml import paths. A piece of the travis config is referenced as vendor/altis/dev-tools/travis/altis.yml which does not exist in the repo until composer install is run on the CI - this is too late.

Docs for travis imports: https://docs.travis-ci.com/user/build-config-imports/

There are 2 things to try:

1. Use a repo import eg. humanmade/altis-dev-tools:travis/altis.yml@ref where we calculate ref dynamically if possible
2. Just copy both files to .config

Also need to add a note stating that altis.yml is managed but can be overridden in the file.

Add Rewrite Rules Inspector

Another useful dev tool (that I just needed) is the Rewrite Rules Inspector. We should consider adding this, and possibly integrating into Query Monitor.

Update Dev Tools module for WordPress 5.8

This is just updating the dependencies, not the QA work.

• Bump WP PHPUnit version to 5.8.1

S3 Uploads AWS SDK exception when scanning uploads dir on test runs

It seems as though the AWS SDK is generating an exception when wp_upload_dir() is called and possibly tries to the test-uploads directory before each test. The exception itself seems to be benign but causes tests to report as failed.

Aws\S3\Exception\S3Exception: Error executing "ListObjects" on "https://s3-dev.altis.dev/?delimiter=%2F&prefix=test-uploads%2F&encoding-type=url"; AWS HTTP error: Client error: `GET https://s3-dev.altis.dev/?delimiter=%2F&prefix=test-uploads%2F&encoding-type=url` resulted in a `404 Not Found` response:
404 page not found
 Unable to parse error information from response - Error parsing XML: String could not be parsed as XML
/usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195
/usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:100
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:203
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:174
/usr/src/app/vendor/guzzlehttp/promises/src/RejectedPromise.php:40
/usr/src/app/vendor/guzzlehttp/promises/src/TaskQueue.php:47
/usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:118
/usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:145
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:246
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:223
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:267
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:225
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:267
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:225
/usr/src/app/vendor/guzzlehttp/promises/src/Promise.php:62
/usr/src/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58
/usr/src/app/vendor/aws/aws-sdk-php/src/ResultPaginator.php:127
/usr/src/app/vendor/aws/aws-sdk-php/src/functions.php:52
/usr/src/app/vendor/aws/aws-sdk-php/src/functions.php:69
/usr/src/app/vendor/humanmade/s3-uploads/inc/class-s3-uploads-stream-wrapper.php:594
/usr/src/app/vendor/wp-phpunit/wp-phpunit/includes/abstract-testcase.php:1063
/usr/src/app/vendor/wp-phpunit/wp-phpunit/includes/abstract-testcase.php:1088
/usr/src/app/vendor/wp-phpunit/wp-phpunit/includes/abstract-testcase.php:111
Caused by
GuzzleHttp\Exception\ClientException: Client error: `GET https://s3-dev.altis.dev/?delimiter=%2F&prefix=test-uploads%2F&encoding-type=url` resulted in a `404 Not Found` response:
404 page not found

Steps to reproduce:

1. On a Travis build showing the above errors start a debug build
2. Once SSH'd in run travis_run_before_script && travis_run_script
3. Tests should run but displaying the above errors

Those errors should definitely not be coming up during test runs.

Acceptance criteria:

• Resolve the AWS SDK errors

Create new API and CLI default test suites and helpers for Codeception

From https://github.com/humanmade/product-dev/issues/848

We discussed having two additional test suites defined for the convenience of engineers, which will be geared towards API, and CLI tests.

This includes setting up new test suites, using respective available modules/helpers from wp-browser eg. WPCLI, and making them available to the bootstrap sub-command so they're created by default unless otherwise specified.

Add platform config to Dev Tools

We should add the parsed config to Devtools, would be handy for locally being able to see how it is coalesced, and also in dev environment, but able to see the values, where you can't get a view of the composer.json

Query Monitor not displaying in Cloud Environments

Moving here from a private repo to add to the backlog. cc @jazzsequence

QM displays in the admin, but not on the front end after the v5 upgrade in cloud environments. This has been confirmed on both dev and staging environments (although it's worth noting that QM wasn't displaying in the admin either on staging prior to v5).

Based on the symptoms and feedback from @johnbillion, it's most likely related to a fastcgi_finish_request call. Note, too, that this is not related to the Altis upgrade, per se, as the platform-test environment does not have this issue.

If Query Monitor's big block of HTML after the closing tag is not present but its toolbar menu is showing and blank, this is usually caused by something calling fastcgi_finish_request() which causes the request output to be flushed before QM outputs its HTML. Example here: Chassis/Chassis_XHGui#19

This issue has not been able to be replicated on local environments.

Acceptance criteria:

• Query Monitor output is properly displayed in the admin and front end for logged in users in cloud environments

PHPUnit does not clear cache when run on a project

Steps to reproduce:

1. Run PHPUnit at the project root.
2. Return to the site to perform any task.
3. WordPress internal CSS and JS 404s and loads from example.org and the main site is Test Blog instead of whatever the site name should be.

Note: Flushing the cache solves the issue.

Expected behavior:

• Site operates normally at local server domain that exists in the database.
• Cache is cleared appropriately

Acceptance criteria:

• ...

Enhance Codeception command with configuration options

@shadyvb commented on Thu Dec 02 2021

Codeception command introduced in #242 could use some configuration parameters to customize the default configuration introduced by the zero-config setup, eg:

• Allow specifying multiple test directories.
• Specifying a custom default browser.

It'd be nice to go through the current generated default config and decide which options can be useful to make available via the Altis config in composer.json.

Simplify import process for Cloud -> Local data imports

Importing data from cloud environments into your local environment is a bit complex currently. We want to simplify this down to improve the user experience.

@joehoyle has existing work started at humanmade/altis-local-server#345 but this was for Local Server rather than generically across our local environments.

As a developer, I want to import cloud exports with a single, simple command so that I don't need to read long documentation every time.

• Running a single command (with a parameter of an export file) imports the exported data
• Database is imported and automatically search-replace'd
• Uploads are imported (i.e. via import-uploads for Local Server)
• Command works across both Local Server and Local Chassis
• Command supports backups containing data-only, uploads-only, or both
• Command outputs useful help text if used without export file parameter
• Command outputs only machine-readable output when passing --porcelain

Add documentation

Currently have no docs, but we should add some fo sho.

PHPUnit options delimiter bug when passing options to PHPUnit

Passing options to PHPUnit doesn't always work as expected, the docs state:

https://www.altis-dxp.com/resources/docs/dev-tools/testing-with-phpunit/

To pass any of the supported command line options to PHPUnit you need to add them after the options delimiter --. For example:

Running tests in a specific directory.

composer dev-tools phpunit -- content/themes/custom-theme/tests

Running tests with code coverage and junit reports.

composer dev-tools phpunit -- --coverage-xml coverage --log-junit junit.xml

The full list of PHPUnit command line options is available here or you can run composer dev-tools phpunit -- --help.

Steps to reproduce:

1. From the terminal of a Altis install with the server running composer server start
2. Run composer dev-tools phpunit -- --version

I expected to see the version of PHPUnit echo'd to the terminal

Instead I see the version of Composer: Composer version 1.10.16 2020-10-24 09:55:59

❯ composer dev-tools phpunit -- --version
Composer version 1.10.16 2020-10-24 09:55:59

I expected to see the same as when running PHPUnit directly from the /vendor folder:

❯ ./vendor/bin/phpunit --version
PHPUnit 7.1.5 by Sebastian Bergmann and contributors.

---

This is also reproducable with the docs example: composer dev-tools phpunit -- --help

Running this results in:

❯ composer dev-tools phpunit -- --help
Usage:
  dev-tools [options] [--] <subcommand> [<options>]...

Arguments:
  subcommand                     phpunit
  options                        

Options:
      --chassis                  Run commands in the Local Chassis environment
  -h, --help                     Display this help message
  -q, --quiet                    Do not output any message
  -V, --version                  Display this application version
      --ansi                     Force ANSI output
      --no-ansi                  Disable ANSI output
  -n, --no-interaction           Do not ask any interactive question
      --profile                  Display timing and memory usage information
      --no-plugins               Whether to disable plugins.
  -d, --working-dir=WORKING-DIR  If specified, use the given directory as working directory.
      --no-cache                 Prevent use of the cache
  -v|vv|vvv, --verbose           Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Help:
  Run a dev tools feature.
  
  To run PHPUnit integration tests:
      phpunit [--chassis] [--] [options]
                                  use `--` to separate arguments you want to
                                  pass to phpunit. Use the --chassis option
                                  if you are running Local Chassis.

This is the equivalent output of composer dev-tools --help which is the dev-tools help and not the phpunit help

Acceptance criteria:

• ...

Add staging to default travis branches list

Would be best to more research on this first before proceeding, seems to be common at Human Made at least.

e2e testing tool

I'm currently experimenting with e2e testing in a project. It started with some sort of automation that let me do some boring and long tasks quickly but I ended up using Jest and WordPress e2e utils.

I'd like to propose the introduction of a dev tools command (something that Rob has already thought about) in Altis that lets you run e2e tests (if there's any) with composer dev-tools e2e or similar.

There are a few considerations before working on this feature:

• The tests must be done in a different instance of Local Server. It's better not to use the same instance where you usually work as it could fill the DB easily and writing tests can be very slow if you have a production DB locally. Additionally, the tests can throw errors due to conflicts with the usual instance.
• In order to create a different instance, Local Server might need to accept an extra parameter like --name=e2e
• Running the tests and activating all the plugins, creating users/posts needed to run those tests can take a lot of time so having an initial DB would be very handy. This can be imported via WP CLI when the tests start running. It's up to the developers to provide that .sql file.
• Everything should play nicely with Travis. We may want to integrate some Travis images in Docker Hub but I haven't done any research on this and I'm not sure if we'll really need it.
• It can work similarly to PHPUnit, searching in similar folders to find test files.
• It can act as a wrapper for [@wordpress/scripts npm package](https://www.npmjs.com/package/@wordpress/scripts). This set of scripts offers commands to launch the e2e tests along with [@wordpress/e2e-test-utils package](https://www.npmjs.com/package/@wordpress/e2e-test-utils).
• It could provide an extensible Jest/bootstrap config. WordPress Scripts also provides one in @wordpress/scripts/config/jest-e2e.config. Altis Jest config can extend that one and call a bootstrap.js defined by default.

I've created a little Gist that can serve as a guide of what files are needed to run the e2e scripts: https://gist.github.com/igmoweb/1d64f820a45b61036022b1270b5c24e3

PHPUnit not clearing all tables properly

On a current client, we are creating sites as fixtures for some complicated functionality tests, but they are not getting cleared from the database upon test startup. I now have over 20 site tables in the test database and it would be really great if Altis would just clear any and all tables with the tests prefix.

Flamegraph JS causes errors on all unauthenticated pages

The flamegraph JS is always loaded regardless of whether it's used or whether QM is even loaded. This causes JS errors on every page.

Fix here: humanmade/aws-xray#43

Add a filter when PHPUnit bootstrap is loaded

Dev Tools load two different bootstrap from inc/phpunit/bootstrap.php:

1. Custom one (if exists) in .config/tests-bootstrap.php
2. WP one in vendor/wp-phpunit/wp-phpunit/includes/bootstrap.php

The problem comes when you try to include your own test cases that will serve as a foundation for tests in your project. The WP bootstrap loads first wp-settings.php (so it fires muplugins_loaded, plugins_loaded and so on) and then it loads all the WP test cases. After that, there's no filter so you cannot include your own files even by using hooks.

There are some ways to do it: requires or by using namespaces but the latter one is sometimes time-consuming if the codebase is old so I was thinking if it would be possible to add some action at the end of the Altis bootstrap, like do_action( 'altis.dev-tools.phpunit_boostrap_loaded' )

Add default Dependabot config if one doesn't exist

On install we should add a default Dependabot config if there isn't an existing one.

It should be set up to create Pull Requests whenever a dependency should be updated according to its version constraint. This will ensure developers are notified and are given an easy system to update dependencies as soon as they are released.

In addition this will mean that teams can more easily rely on only using composer install locally to further ensure that their dependencies are in sync with each other.

Add ability to set dev tools editor per-user

In #25, we added the ability to set your editor in a constant. We want to add the ability to set this per-user, likely in a cookie via some UI.

Switch XRay flamegraph to icicle graph to match Vantage

Way back, we switched the flamegraph in Vantage to an icicle graph and redid all the UI. But we never updated the Dev Tools version to use the same components or UI.

This should be updated to use this for consistency, and because the flamegraph provides a better user experience generally. One option would be to publish the component from Vantage to npm and import that in.

Dependabot can't resolve your PHP dependency files

Dependabot can't resolve your PHP dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Your requirements could not be resolved to an installable set of packages.
  Problem 1
    - The requested package wp-phpunit/wp-phpunit 5.4.2 exists as wp-phpunit/wp-phpunit[3.7.0, 3.7.1, 3.7.10, 3.7.11, 3.7.12, 3.7.13, 3.7.14, 3.7.15, 3.7.16, 3.7.17, 3.7.18, 3.7.19, 3.7.2, 3.7.20, 3.7.21, 3.7.22, 3.7.23, 3.7.24, 3.7.25, 3.7.26, 3.7.27, 3.7.28, 3.7.29, 3.7.3, 3.7.30, 3.7.31, 3.7.32, 3.7.33, 3.7.4, 3.7.5, 3.7.6, 3.7.7, 3.7.8, 3.7.9, 3.8.0, 3.8.1, 3.8.10, 3.8.11, 3.8.12, 3.8.13, 3.8.14, 3.8.15, 3.8.16, 3.8.17, 3.8.18, 3.8.19, 3.8.2, 3.8.20, 3.8.21, 3.8.22, 3.8.23, 3.8.24, 3.8.25, 3.8.26, 3.8.27, 3.8.28, 3.8.29, 3.8.3, 3.8.30, 3.8.31, 3.8.32, 3.8.33, 3.8.4, 3.8.5, 3.8.6, 3.8.7, 3.8.8, 3.8.9, 3.9.0, 3.9.1, 3.9.10, 3.9.11, 3.9.12, 3.9.13, 3.9.14, 3.9.15, 3.9.16, 3.9.17, 3.9.18, 3.9.19, 3.9.2, 3.9.20, 3.9.21, 3.9.22, 3.9.23, 3.9.24, 3.9.25, 3.9.26, 3.9.27, 3.9.28, 3.9.29, 3.9.3, 3.9.30, 3.9.31, 3.9.4, 3.9.5, 3.9.6, 3.9.7, 3.9.8, 3.9.9, 4.0.0, 4.0.1, 4.0.10, 4.0.11, 4.0.12, 4.0.13, 4.0.14, 4.0.15, 4.0.16, 4.0.17, 4.0.18, 4.0.19, 4.0.2, 4.0.20, 4.0.21, 4.0.22, 4.0.23, 4.0.24, 4.0.25, 4.0.26, 4.0.27, 4.0.28, 4.0.29, 4.0.3, 4.0.30, 4.0.4, 4.0.5, 4.0.6, 4.0.7, 4.0.8, 4.0.9, 4.1.0, 4.1.1, 4.1.10, 4.1.11, 4.1.12, 4.1.13, 4.1.14, 4.1.15, 4.1.16, 4.1.17, 4.1.18, 4.1.19, 4.1.2, 4.1.20, 4.1.21, 4.1.22, 4.1.23, 4.1.24, 4.1.25, 4.1.26, 4.1.27, 4.1.28, 4.1.29, 4.1.3, 4.1.30, 4.1.4, 4.1.5, 4.1.6, 4.1.7, 4.1.8, 4.1.9, 4.2.0, 4.2.1, 4.2.10, 4.2.11, 4.2.12, 4.2.13, 4.2.14, 4.2.15, 4.2.16, 4.2.17, 4.2.18, 4.2.19, 4.2.2, 4.2.20, 4.2.21, 4.2.22, 4.2.23, 4.2.24, 4.2.25, 4.2.26, 4.2.27, 4.2.3, 4.2.4, 4.2.5, 4.2.6, 4.2.7, 4.2.8, 4.2.9, 4.3.0, 4.3.1, 4.3.10, 4.3.11, 4.3.12, 4.3.13, 4.3.14, 4.3.15, 4.3.16, 4.3.17, 4.3.18, 4.3.19, 4.3.2, 4.3.20, 4.3.21, 4.3.22, 4.3.23, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 4.3.8, 4.3.9, 4.4.0, 4.4.1, 4.4.10, 4.4.11, 4.4.12, 4.4.13, 4.4.14, 4.4.15, 4.4.16, 4.4.17, 4.4.18, 4.4.19, 4.4.2, 4.4.20, 4.4.21, 4.4.22, 4.4.3, 4.4.4, 4.4.5, 4.4.6, 4.4.7, 4.4.8, 4.4.9, 4.5.0, 4.5.1, 4.5.10, 4.5.11, 4.5.12, 4.5.13, 4.5.14, 4.5.15, 4.5.16, 4.5.17, 4.5.18, 4.5.19, 4.5.2, 4.5.20, 4.5.21, 4.5.3, 4.5.4, 4.5.5, 4.5.6, 4.5.7, 4.5.8, 4.5.9, 4.6.0, 4.6.1, 4.6.10, 4.6.11, 4.6.12, 4.6.13, 4.6.14, 4.6.15, 4.6.16, 4.6.17, 4.6.18, 4.6.2, 4.6.3, 4.6.4, 4.6.5, 4.6.6, 4.6.7, 4.6.8, 4.6.9, 4.7.0, 4.7.1, 4.7.10, 4.7.11, 4.7.12, 4.7.13, 4.7.14, 4.7.15, 4.7.16, 4.7.17, 4.7.2, 4.7.3, 4.7.4, 4.7.5, 4.7.6, 4.7.7, 4.7.8, 4.7.9, 4.8.0, 4.8.1, 4.8.10, 4.8.11, 4.8.12, 4.8.13, 4.8.2, 4.8.3, 4.8.4, 4.8.5, 4.8.6, 4.8.7, 4.8.8, 4.8.9, 4.9.0, 4.9.1, 4.9.10, 4.9.11, 4.9.12, 4.9.13, 4.9.14, 4.9.2, 4.9.3, 4.9.4, 4.9.5, 4.9.6, 4.9.7, 4.9.8, 4.9.9, 5.0.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.1.0, 5.1.1, 5.1.2, 5.1.3, 5.1.4, 5.1.5, 5.2.0, 5.2.1, 5.2.2, 5.2.3, 5.2.4, 5.2.5, 5.2.6, 5.3.0, 5.3.1, 5.3.2, 5.3.3, 5.4.0, 5.4.1, dev-master, dev-tree-3.7, dev-tree-3.8, dev-tree-3.9, dev-tree-4.0, dev-tree-4.1, dev-tree-4.2, dev-tree-4.3, dev-tree-4.4, dev-tree-4.5, dev-tree-4.6, dev-tree-4.7, dev-tree-4.8, dev-tree-4.9, dev-tree-5.0, dev-tree-5.1, dev-tree-5.2, dev-tree-5.3, dev-tree-5.4] but these are rejected by your constraint.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Contribute editor configuration upstream

The editor configuration tools we've been working on can be contributed back upstream; John has indicated he'd be willing to add them: #25 (comment)

Follow-on from #16

Document how to authenticate with Docker on Travis

Due to Docker's new rate limits we need to inform clients and partners how to set up Travis up to use an authenticated account to avoid the problem.

Thanks to @nathanielks - the steps are:

• Create an account on Docker Hub
• Create an Access Token in Docker Hub
• Add an environment variable named DOCKER_USERNAME to your Travis environment with the value of your Docker Hub username.
• Add an environment variable named DOCKER_PASSWORD to your Travis environment with the value of the Access Token created before.
• Add a before_install section of your .config/travis.yml file:

before_install:
  - echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin

• Commit and push these changes to a branch to trigger a build.

We can also suggest they make it a pro account to avoid the rate limits unless we decide to suggest people make a support request to us to set this up using the HM account.

Updating from 8.0.2 to 8.0.3 breaks phpunit command

Steps to reproduce:

1. From a repository using v8.0.2 of this module where the phpunit subcommand is currently working,
2. Run composer update to pull in latest v8.0.3

• Observe - Upgrading wp-phpunit/wp-phpunit (5.7.2 => 5.7.4) in composer output: probably the culprit?

2. Run composer dev-tools phpunit to execute tests with dev-tools v8.0.3

Expected to see Unit Tests executed. Actually see this error:

> composer dev-tools phpunit

Error: The PHPUnit Polyfills library is a requirement for running the WP test suite.
If you are trying to run plugin/theme integration tests, make sure the PHPUnit Polyfills library (https://github.com/Yoast/PHPUnit-Polyfills) is available and either load the autoload file of this library in your own test bootstrap before calling the WP Core test bootstrap file; or set the absolute path to the PHPUnit Polyfills library in a "WP_TESTS_PHPUNIT_POLYFILLS_PATH" constant to allow the WP Core bootstrap to load the Polyfills.

If you are trying to run the WP Core tests, make sure to set the "WP_RUN_CORE_TESTS" constant to 1 and run `composer install` before running the tests.
Once the dependencies are installed, you can run the tests using the Composer-installed version of PHPUnit or using a PHPUnit phar file, but the dependencies do need to be installed whichever way the tests are run.

This occurs both locally, and in CI. Using Composer v2 and PHP 7.4 on host OS as well as within Altis VM (local-server).

Acceptance criteria:

• composer dev-tools phpunit runs unit tests without error

Two packages are abandoned

Describe the bug
When using the V3 version of altis/altis composer warns of two dependency packages which are abandoned.

Package johnbillion/query-monitor is abandoned, you should avoid using it. Use wpackagist-plugin/query-monitor instead.
Package phpunit/phpunit-mock-objects is abandoned, you should avoid using it. No replacement was suggested.

Our current project file includes the following:

...
    "require": {
        "altis/altis": "^3.0",
 ...

To Reproduce
I haven't attempted to replicate this with a clean install... so ping me in slack if a reference to the build error in Altis Dashboard would be useful.

Expected behavior
No dependancy warnings when using altis/altis v3 or altis-dev-tools

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: OSX 10.15.5
• Composer: 1.9.3
• PHP version: 7.4.7

Default Travis build does not allow changing Node version

Steps to reproduce:

1. Add node: 12 to travis config
2. Scripts use default node version

Setting the node version should be reflected in the runtime for scripts & install steps.

Acceptance criteria:

• Remove the matrix setting
• Add explicit setting for PHP version at 7.2

phpunit does not output to the screen until complete run is completed

When running the command composer dev-tools phpunit the output is only displayed on the screen after all the test suite is completed.

This creates a problem in Travis CI runs where the test suite takes more than 10 minutes and travis ends the job since it does not receive any output for 10 minutes.

This can also be reproduced locally when running composer dev-tools phpunit only shows the output after all tests are completed.

Enable by default on dev/staging environments

Maybe staging too?

Configure clickable traces by default

https://querymonitor.com/blog/2019/02/clickable-stack-traces-and-function-names-in-query-monitor/

We have enough integration to be able to enable this by default.

PHPUnit directories config option does not support wildcards

Noticed while debugging a client project's Travis CI build.

We're checking the directories exist which doesn't really work for wildcards, could try a glob instead or just leave the configuration up to the developers to get right.

Add pane for Cloud X-Ray

It would be good to show the XRay profile in a dev-tools custom panel. Bonus points if we can get the flamgegraph in there.

Travis CI fails with a fresh Altis v8 install

Steps to reproduce:

1. Create a new Altis v8 install from humanmade/altis-skeleton
2. Push the code to a GitHub repo
3. Open Travis CI and navigate to the project repo

Travis CI job fails with the following error:

0.47s$ composer local-server start
Starting...
unknown shorthand flag: 'f' in -f

The preceding composer install installs packages successfully (below is a brief summary of the Travis environment)

Setting environment variables from .travis.yml
$ export COMPOSE_HTTP_TIMEOUT=360
$ export ES_MEM_LIMIT=2g

0.02s$ phpenv global 7.4 2>/dev/null
7.4 is not pre-installed; installing

0.87s$ composer self-update
Upgrading to version 2.1.5 (stable channel).

$ php --version
PHP 7.4.15 (cli) (built: Feb  7 2021 09:15:09) ( ZTS )

$ composer --version
Composer version 2.1.5 2021-07-23 10:35:47
3.76s$ nvm install v12
0.47s$ nvm use v12

7.23s$ composer install
0.47s$ composer local-server start
Starting...
unknown shorthand flag: 'f' in -f

Running Travis CI in debug mode with SSH:

Legacy "Docker Compose" syntax works as expected:

docker-compose -f

New "Docker Compose" syntax does not work as expected:

docker compose -f
unknown shorthand flag: 'f' in -f

Thus, I think that as Travis CI is using the legacy docker-compose command here it's related to humanmade/altis-local-server#289

Acceptance criteria:

• ...

Document how to authenticate with GitHub on Travis for private deps

For projects using Git submodules referenced by the SSH URL then it requires adding multiple deploy keys or setting up a user key with access to everything needed.

Similar to #122

Reference: https://docs.travis-ci.com/user/private-dependencies/

Expose local environments publicly

Developers often need a way to expose their local environments publicly, for testing purposes.

We can provide some documentation, and/or a command to expose a local environment publicly using ngrok, via a dedicated proxy container.

Update the codeception command to use a path rather than a module name

Currently the codecept command can take an option -m to specify a module to run tests. We should change this to a generic -p|--path option to optionally specify a tests directory.

Acceptance criteria:

• Default tests path is /tests in the project root
• Default tests path can be set via -p|--path option
• If codeception.yml exists in path skip config generation and call codecept
• If _supports directory exists in path use that, else default to _supports directory in this module
• If _data directory exists in path use that, else default to _data directory in this module
• If _envs directory exists in path use that else default to _envs directory in this module
• _outputs directory remains as-is in the tests path

Conflict with xray

XRay using a shutdown handling causing fastcgi_finish_request so the output from QM never appears, need to get them working together.

QM shows a db.php warning.

Using this repo locally is causing Querty Monitor to show the warning:

Extended query information such as the component and affected rows is not available. Query Monitor was unable to symlink its db.php file into place.

Including vendor/johnbillion/query-monitor/wp-content/db.php as a drop-in should resolve this.

Altis Config appers twice in Query Monitor side panel

Right at the top and far down close to the bottom of the side panel.

Document how to enable/disable

This may not be enabled correctly.

Show "Altis" as component instead of "Unknown"

Requires changes to QM, see johnbillion/query-monitor#450

Disable certain docker images from loading in the CI to improve the time required to build the project when running tests

When running the CI tests we need to set up local-server and that by-default pulls all the services even though they are not used in the CI environment and that adds up to the time required to run the tests in the CI.

For eg I believe - kibana, X-Ray, Cavalcade, mailhog can be disabled by default.
Also based on project requirements things like Pinpoint, Cognito, tachyon can be disabled as well.

For now, we are disabling by forking the docker-compose.yml from the local-server in the project and copying this file in the CI environment just before starting the local-server.

But it would be nice to have a way to do this without needing to maintain a forked docker-compose file.

I see we are looking to generate the docker-compose.yml file in PHP (humanmade/altis-local-server#159) hoping that will help with this?

Enhance Codeception command

Some enhancement ideas spun out of the work on https://github.com/humanmade/product-dev/issues/848 to improve the process of writing and running Codeception tests, and the current documentation pull-request at #242 assumes these are carried out.

• The command should detect existing test suites in the specified directory, and run them one after the other, but not in the same session, eg: multiple remote calls to the codecept command.
• The command should detect whether the current selected suite would need a browser ( detecting use of WPWebDriver module ) and run it automatically, removing the requirement of using the -b flag.
• WPCLI environment variable is respected just like the headers of acceptance test requests, ie: WPBROWSER_HOST_REQUEST to switch to the test database.
• Create a new bootstrap subcommand of codecept command to create/copy all (or subset) of test suite configurations. eg: composer dev-tools codecept bootstrap acceptance,functional

Make travis.yml generation optional

For some projects, there's no need for a CI config, as there are no tests. Deleting the file from the repo fixes it, but the file gets created again every time local server runs.
A workaround would be to .gitignore it.
It would be preferable to make it optional

Document setting up codeception tests for projects

With the new codeception command, we can run tests for Altis packages, but projects can use the testing configuration to run project-specific tests too, so we need to document how to set it up, and how to write different styles of tests as per codeception/wp-browser documentation.

Actions:

• Add docs for setting up project-specific tests
• Add docs around different types of tests and link to Codeception/wp-browser docs

Document use of dedicated deploy user for private dependencies

On HM project's we use a dedicated user account to provide authentication for CI platforms to access private dependencies and to check out submodule referenced by SSH URLs.

https://docs.travis-ci.com/user/private-dependencies/#user-key

We should document this approach on the Continuous Integration docs page.

Provide GitHub Actions sample config within docs

Related to https://docs.altis-dxp.com/nightly/dev-tools/continuous-integration/

We recommend, and generate config for, Travis as the recommended CI env, is there something that prevents us from suggesting GitHub Actions as an alternative CI env, with basic docs and examples ?

Document how to use X-Ray flamegraphs

Currently, we have no documentation anywhere (AFAIK) on how to use X-Ray or how to read flamegraphs. They're a useful tool, so would be good to add this.

altis/dev-tools as a dependency causes unnecessary non-dev dependencies

If you follow the basic Getting Started instructions for an Altis project you end up with a bunch of testing libraries in your Composer dependency graph when installing the site for production use with composer install --non-dev.

This includes phpunit/phpunit, wp-phpunit/wp-phpunit, and others.

The reason is that this altis/altis meta package has altis/dev-tools as a non-dev dependency, which in turn has a non-dev dependency on phpunit/phpunit and wp-phpunit/wp-phpunit.

I think the problem is that altis/dev-tools should be a dev dependency rather than a dependency.

Thoughts?