Fixing flaky parallel tests

Fixing flaky parallel tests

I'm currently working on a large Laravel project with about 1.900 tests. Unfortunately, running those tests in sequence takes about ten minutes, and nobody has time for that. Fortunately, Laravel comes with a parallel test runner (https://github.com/paratestphp/paratest), which makes use of all available cores on your machine and cuts the time down to about 2:30 minutes. After I've written about 1.850 tests, running the whole suite sometimes resulted in tests failing. When running those failing tests, they're passing. I was stunned, not knowing where to start to fix this annoyance. Consulting the PHPUnit docs (https://phpunit.readthedocs.io), I came across the processIsolation attribute, which resides in the phpunit.xml configuration file. The documentation reads

This attribute configures whether each test should be run in a separate PHP process for increased isolation.

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    backupGlobals="false"
    backupStaticAttributes="false"
    bootstrap="vendor/autoload.php"
    colors="true"
    convertErrorsToExceptions="true"
    convertNoticesToExceptions="true"
    convertWarningsToExceptions="true"
-   processIsolation="false"
+   processIsolation="true"

This simple change seems to fix my problem; now, all tests pass all the time.

Update: Although setting processIsolation to true fixes tests flaking out, the run time is about ten minutes and therefore utterly useless. I found out that running ./vendor/bin/paratest instead php artisan test --parallel really fixes my problems. I'm currently on the brink of submitting an issue to the Laravel repository, but it isn't effortless to create a proof of concept to attach to the submission.