facile-it/paraunit

Run PHPUnit tests in parallel

paraunit's Issues

Windows support

To support windows, those stuff is needed:

• Add AppVeyor as CI
• Prepend PHP to every execution, since the shebang is not useful
• use DIRECTORY_SEPARATOR when generating filenames for logs and coverage
• Backport fixes to 0.6.x?

CONTRIBUTING.md

I think it could turn useful to have a CONTRIBUTING.md file.
Nothing fancy just some basic tasks to run before submitting a PR and maybe the general guidelines.

More so, Paraunit docker container comes packed with all the tools required to follow these rules, particularly (but not only):

• phpcbf
• phpcs

It would be even better to provide a custom version of the of such tools, even a contributing.sh with multiple tasks inside. Something that a developer can run before submitting his contribution.

---

TO DO

• Choose a coding standard (full PSR2)
• Sync scrutinizer with the new coding standard
• Add git pre-commit hook
  • /src directory
  • /tests directory (need to fix non camel-case method names)
• Add a CONTRIBUTING.md

Allow Symfony 3.1 components

As from title

Abnormal termination with empty output

When an abnormal termination with no output is being reported, we should handle that with a specific message, or it would be unclear what happened.

Report when fail to load coverage data

As in title.

Fatal errors are different on HHVM & PHP7

https://travis-ci.org/facile-it/paraunit/jobs/81814347

PHP7 and HHVM handle fatal errors as errors.... We should fix the parsing (and the build!)

[Refactor] Changing parser structure

Right now, we have a lot of duplication between Error\Failure\Warning parsers; the same is true for the process interface, where we have a lot of getter/setters/hassers. It then pour out to the printers, where we have a lot of duplication in the containers of the parser results.

Refactor is needed to boil it down to just one type of class with multiple identifies (constants?) on all 3 sides of this issue.

Proposed path:

• move OutputContainers into corresponding parsers
• let parsers accumulate output into OutputContainer
• make ProcessOutputParser a dependency of FinalPrinter
• expose parsers (iterable) through ProcessOutputParser
• expose OutputContainer through parsers
• use OutputContainer init value as a tag on the console formatter
• test that each tag like that is added through the ConsoleFormatter
• test unknown output for edge cases (PHP strict standards, notice, warnings...)

New course of action:

• ProcessFactory adds the option for JSON logging to a temp dir (/dev/shm?)
• Update printers
• On process term, parse the JSON log for test results
• Specific chained parsers for each type of outcome that should be handled
• On process term, parse the JSON log for test error messages
• skip\handle suite start/end
• Cycle the log, 2 at a time
• for each couple, parse and add the result
• if latter is missing, signal an abnormal termination into a specific function
• clean up temp dir at engine startup
• test all outcomes (ok, errors, fails, fatal, segfaults)
• remove unused parsers
• add stack trace and function name printing to FinalPrinter

Add parsing for skipped tests

PHPUnit (at least in version 5) prints output when a test is skipped with a message. We should parse them!

• Parse skipped messages
• Parse incomplete messages (they exists?)
• Discover if this is also present on PHPUnit 4

Sqlite database locked exception non parsed for retry

Using sqlite as database with parallel functional tests I found this error that is not matched to retry

SQLSTATE[HY000]: General error: 5 database is locked (SQL: insert into "wallets" ("name", "user_id", "updated_at", "created_at") values (test_wallet, 1, 2015-11-25 21:07:38, 2015-11-25 21:07:38))

Phar Archive

I think that we should distribute paraunit as a phar archive in order to allow users to simply wget it from our site and than use it without the need of anything else than php

Update documentation about new coverage feature

Documentation needs info about the new coverage command, about its usage and requirements.

--group and --exclude-group support

Hi, would be nice to have --group and --exclude-group support

Use PHPUnit test result logs instead of stdout parsing

PHPUnit has support for test result logging (more info here). Paraunit could use these instead of parsing stdout which may contain garbage data (eg. var_dumps).

Segmentation fault attempt

@fntlnz Do you know if PHP7 and HHVM still output

segmentation fault (core dumped)

In case of a segmentation fault?

For what i can see after commit 0dfc77b the output seem to be quite different.

HHVM:

Fatal error: String length exceeded 2^31-2: 18446744071626162096 in [...]

PHP7:

Fatal error: Out of memory (allocated 4194304) (tried to allocate 799999999999952 bytes) in [..]

Abstract test cases should not be executed

Looks like we should skip abstract classes in the test folder.

Few examples in Symfony
https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Validator/Tests/Validator/AbstractTest.php
https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Form/Tests/AbstractFormTest.php

Abnormal Terminations (fatal Errors, Segfaults) output:
1) UNKNOWN -- log not found
Abnormal termination -- complete test output:
Class '/home/travis/build/laboro/dev/package/platform/src/Oro/Bundle/TranslationBundle/Tests/Functional/ImportExport/Strategy/AbstractTranslationImportStrategyTest' could not be found in '/home/travis/build/laboro/dev/package/platform/src/Oro/Bundle/TranslationBundle/Tests/Functional/ImportExport/Strategy/AbstractTranslationImportStrategyTest.php'.

PHP < 5.5 support.

Today I've broken the build twice (4d31ec1 & 2e5b5ee) because I was using [] instead of array() and the ::class static shortcut for Prophecy.

PHP 5.3 is alread dead, and 5.4 is already on extended support, and it will dead 1 month from now: http://php.net/supported-versions.php

I propose to push the minimum support to 5.5, to encourage the usage of the supported version.

Github pages doc

I was wondering if it's not time to build a documentation website for this project.

Something like this would be very nice: https://facebook.github.io/watchman/

Abandon the segfault simulation

After a quick chat on Twitter last night, @taueres suggested to me to use exit(139) as a way to emulate test segfaults in Paraunit's testsuite. Since we ditched output parsing in favor of JSON logs, this should be feasible.

Test filtering needs to fully read from PHPUnit xml config file

With commit 75cfc1f, Paraunit started to pull up files reading the xml config file. It needs to be improved:

• support prefix elements
• support suffix elements
• support exclude elements
• support file elements
• accept test suite filtering from CLI

Load XML config with default name

When using the --configuration option, we are forced to specify the file name. We should emulate PHPUnit's behaviour, append the default filename (phpunit.xml.dist) if it's not specified.

Make logo printer optional or configurable

I would like to make the shark logo optional or configurable with the stable release. It was nice for the unstable, but I think it may look "personalized" for a software that we like to distribute.

Add support for more options for test selection

I would like to support more option to select test, other than testsuite definition. Please, suggest what you would like to see supported.

Options:

• --group CLI parameter
• ... ?

Refactor FinalPrinter

It should be splitted into multiple classes, listeners to the same event, with the right priority.

--stop-on-error stop-on-failure implementation

With most of the execution option behing ralyed tho the child processes here #57
it would be an interesting feature to have Paraunit handle further execution option, especially the ones that could make sense in the global execution or are non-relevant for the child processes.

In this case the the focus of this issue is on:

--stop-on-error 

To avoid misunderstanding and promote ease-of-use the behavior should be as similar as possible to the phpunit documentation:

--stop-on-error Stop execution upon first error.
--stop-on-failure Stop execution upon first error or failure.
--stop-on-risky Stop execution upon first risky test.
--stop-on-skipped Stop execution upon first skipped test.
--stop-on-incomplete Stop execution upon first incomplete test.

Contrary to phpunit, paraunit execution is parallel by definition and for this reason it's hard to actually stop all running process if on of them return an error.

In this case it could be a best effort and stop the spawning of new processes, disregarding the output of running ones and printing all accumulated output.

With this approach the original idea behind the --stop-on-error should be maintained as the execution will terminate with an ASAP policy.

Code coverage doesn't work with PHPDbg

After testing #59 I found that the PHPDbg support is lacking:

• PHPDbgBinFile::isAvailable() fails: probably file_exist doesn't have enough permissions
• PHPUnit throws SebastianBergmann\CodeCoverage\RuntimeException, probably when merging the coverage, if xdebug is missing

PHP Warnings marked as F

I've noticed that php warnings are marked as F but build return code is 0. Why not mark them as W or something similar ?

Drop unsupported language/package versions

I would like to bump the supported versions for those deps with the 1.0 stable release:

• PHPUnit ^6.0 (4.x drops support 2016-02-27, cross support is too hard)
• Symfony ^2.7 || ^3.0
• PHP ^7.0 (due to lack of support from PHPUnit 6)

This means also a few optimizations in the works:

• support PHPUnit 6 (partially done with #86 )
• use ::class everywhere
• use array short syntax everywhere
• use scalar and return typehint everywhere
• use strict_types=1 everywhere
• check for PHPUnit deprecations (setExpectedException, ...)
• check for Symfony deprecations (maybe using symfony/phpunit-bridge)
• drop Symfony/HttpKernel and the RegisterListenersPass workaround
• disable skipping for RunnerTest::testWarning
• remove check about presence of JSON_PRETTY_PRINT, it's there since 5.6.1
• remove fallback versions of assertFile[Not]Exists
• check that PHPUnit's options are correctly aligned with the 6.x version

PHP7 (and HHVM??) In travis

Since PHP 7 is in the process of being released IMHO we should consider start supporting it right now.

A first step to do this could be to tell travis to build the project against it as described here

Are we considering to support this under HHVM?

Print what coverage driver is being used

We should write what driver is in use before launching all the tests

Too much disk space occupied

Yesterday I tried launching the coverage command over a very large codebase. The execution failed due to not having space to save the logs and the coverage results. We should delete the logs and the files right after reading them, to save on space while executing the tests.

Improve --debug output

As it is, the output generated by --debug option is not very clear, it needs to be improved.

Long time consuming functional tests blocking paraunit end

Paraunit execution remains blocked waiting for long functional tests that are running at the end.
I'm working on a solution to eliminate the waiting time for these last long tests.

The idea is to extract the process stack creation into a service that can use some different strategies to "build" the stack.
One strategy (the default one) is the actual strategy, the other one creates an index with the timing of execution of every single test class and builds a stack based on this index.
With this strategy the longer tests will be executed for first, optimizing the execution and eliminating the long final wait time.

I'm thinking to use different adapters (File, db, others ..) to store che "index" and i would use some sort of configuration to drive it.
I'm thinking about use dotEnv for this purpoose (actually i'm using the command)

Can this solution be usefull?

Here is my forked project development

[Proposal] Paraunit configuration with dotEnv

I think that paraunit needs some sort of configuration file to set actual and future parameters, for example for portability in CI environments.

I want to propose this implementation using dotEnv feature_dotenv_configuration

Git hook is fired on unrelated branch

I was updating the Paraunit site for documentation, and the phpcs hook fired and failed (the src/ folder was missing!)

We should change the script to "blacklist" the branches relatives to the site

Output containers should ignore empty messages

If an output containers receives a null or emptry string as a message, they should discard them, to avoid spamming in the final output.

The same should be true for skipped and incomplete empty messages, as Skipped Test: or Incomplete Test:, that are triggered when no string is passed to the markTestAsSkipped() or markTestAsIncomplete() functions.

Parallel code coverage

I've started to work to a new feature, a command that produces code coverage with Paraunit.

I'll merge the branch ASAP, and I'll use this issue to track the progress.

• Big refactoring
• First working version
• Support for PHPDBG (PHP7)
• More testing!
• Merge feature-coverage to master
• Create 0.7 milestone and add this issue to it
• Check for syntax before merging the coverage results
• Try to have parallel result merging (recursive + binary approach?)
• Check for performance issues

[...]

• Tag as 0.7 and close this issue!

Support for PHPUnit 5.x

Right now, our composer.json forbids the use of PHPUnit 5.x.
I would like to rise that limit; we need to test if there is any BC break that will impede that.

Colors no longer working

I do not see any colored output using the master branch. Maybe #7 broke something about the console formatters?

License owner and copyright year

License owner and copyright year are missing from the LICENSE file.

https://github.com/facile-it/paraunit/blob/master/LICENSE#L189

Support PHPUnit's warnings

PHPUnit implemented warnings: sebastianbergmann/phpunit#1802

Paraunit should be able to parse them.

Testbench

The goal 1510_paraunit_funfair it's to provide some dockerfiles used for running paraunit against some well consolidated testsuites.

• Monolog
• Swiftmailer
• Symfony2
• Doctrine2
• Zend2
• Laravel

By running:

./testbench/monolog/setup.sh

The script will hijack docker mounting functionality putting paraunit source directly inside monolog vendor.

It will be possible to run paraunit against monolog while playing with the source.

sample output:

                                                    B>                           
                                                   B "Bp                        
.pp..                                              B    9p                      
 "9BBBBBBpp.                                       B      9p                    
    " ""9BBBBBBpp                          .<eeP"B B      .B b                  
           "SANDROBpp              .     B B     B B      )B B                  
              "BFRABBBB>  .<pe6P\B B     B B     B B      $  B     .e           
                 5NICOBBB B     ·B B     B B     B Bqp.  :B  B     $ 4BBpp      
                   BMIKIB B        B     B B     B B   "^Bp  B    ) |BBB"\BBpp. 
                 .BALEBBB """9q.   B"""""B B"""""B B      1p B""""9p BBBBbBBBBBBB
               <BLUCABBBB B    "B  B     B B     B B       B B     9 9BBB< ^P"  
            .6BSERGIOBBBB B666666B B     B B     B B       9 P      7 9BBBBP    

PARAUNIT V0.4 - by Francesco Panina, Alessandro Lai and Shark Dev Team @ Facile.it
....................................................SS..........................
..................................................................E.............
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
.........................................................SSSS...................
...................S.....S....................S.......................SSSSSSSS..
........................................

Execution time -- 00:00:05 

Executed: 68 test classes, 840 tests

ERRORS output:

1) Monolog\Handler\RavenHandlerTest::testUserContext
Undefined index: sentry.interfaces.User

/tmp/monolog/tests/Monolog/Handler/RavenHandlerTest.php:120

1 files with ERRORS:
 RavenHandlerTest.php

6 files with SKIPPED:
 ZendMonitorHandlerTest.php
 GelfHandlerLegacyTest.php
 ElasticSearchHandlerTest.php
 CouchDBHandlerTest.php
 AmqpHandlerTest.php
 MongoDBFormatterTest.php

Docker PHP enviroment

It would be nice to compile PHP from source (using one of the official PHP docker images) rather than relaying to Debian's repository.

This would give us the ability to easily create 2 docker images:

• PHP 7.x (FROM php:7/7-cli)
• PHP 5.6.x (FROM php:5.6/5.6-cli)

and also keep the container cleaner.

what do you think?

Delete composer.phar

The file composer.phar should not be part of the project. It's also quite bit ~1 MB.
I suggest to remote it and add a new .gitignore rule.

Refactor RetryManager into RetryParser

As stated in #7 's comments, we should move the Retry logic into the RetryParser. As a matter of fact, the manager now just parses the output!

Memory and time consumption

Currently, we use PHPUnit plugins to track single test case duration.
Plugins reporting to output will not work with JSON approach in Paraunit

Would be good to have an ability to track single test case memory and time consumption.
Threshold option might filter results not to report all tests
Another option might control test case failure in case test case is slow or a lot of memory used

Support TestDox

I think that would be nice if paraunit can generate the PHPUnit testdox in parallel, what do you think?

Test file path are relative to .xml config file path

I was trying to use Paraunit in a project where the .xml config file isn't in the root dir of the project itself.

This causes an issue: the test suites are defined with relative paths, but they are relative to the config file position. This must be fixed.

When the --configuration parameter is passed to paraunit, the filename should be stripped from the value, and the remaining is to be prepended to every file test path when launching the PHPUnit processes.

Block PHPUnit colors in CLI instead of requiring to disable it in XML config

I've just discovered that you can disable PHPUnit colors in CLI with:

--colors=never

I would like to enforce it on every PHPUnit process that Paraunit launches, to avoid parsing problems and to remove from requirements the disable of colors from the XML config file.

BTW, I need first to investigate from which version of PHPUnit this option is supported... I think I'll open an issue on PHPUnit because this option is undocumented.

Add Hook and lifecycle logic

@Jean85 With commit 9d4bf8e we can now add a lifecycle logic to the runner.

With this little upgrade it will be easier to add any new behavior such:

• Something that need to be executed before the process is started (eg. Event: process.before_start and process.start).
• Something that need to be executed when the engine complete his work (eg. Event: engine.end).

At the present time we only have engine.before_start which is hooked to the output formatter and shark printer but the infrastructure is there!

My proposal is to move some dependency away from the runner so it will be easier to unit test.
As a side note, using events in key areas will improve the system extensibility.

var_dump() leaks to the main output

As from title.

Maybe we should reuse the output redirection?