I think you can remove php 7.4 and 8.0 until there is a new phpunit version that fix it and we use it.

https://github.com/php/php-src/blob/03ba053af0240daeff9fbde4d37de2565df65429/ext/reflection/php_reflection.c#L6521

7.4 should be there. There should always be a run with at least the next minor PHP version in an allowed fail state until that branch reaches late beta or early RC. Right now the 8.0 run can probably go away because most packages have Composer version constraints that block them being installed on 8.0 right now so it's wasted CPU time.

That way someone can proactively work on compatibility issues so that on release day the software is ready to go. Or, you can stick with the Joomla trend of waiting until after the stable release to even start looking at compatibility issues.

Failing is not the problem, but running the test if the testing framework is broken makes no sense.
So we have to update phpunit (if there is a fix).

And that impacts Joomla's CI setup how? The entire point of the documentation commit and noted pull request were to fix an issue PHP core introduced that created problems with PHPUnit and other testing frameworks. Otherwise, by that claim PHPUnit is not compatible with anything 7.1.0 or later, and that's clearly not the case.

There should always be an eager run for next version, or you can just hope and pray that someone has enough time on their hands to compile PHP from a source branch to run testing (hint, nobody will). It's this eager run that has allowed people to address potential compatibility issues with every PHP release since 5.6 and the only thing it is costing is CPU time (and now manpower since people would rather spend their time self managing CI resources instead of partnering with a third party solution provider).

#24989

And yes, I'm advocating for running our own drone, since it is the most complete of the 4-5 CI systems that we have. And most of the work is still to write the tests, not to setup that system.

Most complete is subjective, and we all know what they say about opinions. I've not run into anything I can't do on Travis, and that includes the pretty comprehensive test matrix for the standalone database Framework package to ensure it's getting tested across every supported PHP version and damn near every database version we claim to offer support for across every PHP version we have (and if I had to write Docker images for the 70 build variants that are supported for the 1.x branch (PHP 5.3+, MySQL 5.5 thru 8.0, MariaDB 10.0 thru 10.2, and PostgreSQL 9.1 thru 11.x; SQL Server on 1.x is only tested in AppVeyor) I'd just wipe out the tests directory and say "good luck, chump").

Realistically the only thing you're getting out of self-hosted Drone over a dedicated CI service provider is dedicated hardware, and for the paranoid people you're not letting your source code "leak" to a third party service (which, in the context of Joomla, that concern realistically only applies to the JSST repo). Otherwise, most of the CI setup steps are pretty similar in the end. And I personally think time is better invested on things that have to be done in house instead of creating more in house work for an already limited resource pool.

I didn't say that you couldn't do these things with Travis or Jenkins or any other CI system. What I wanted to say was, that our Drone setup was the one that had the most steps/work/whatever in it. Codestyle for PHP & JS, Unittests for PHP, Tests for JS, Systemtests for PHP and RIPS security tests. To me it was sane to use the system that had matured the most in our main repo and extend that, instead of migrating the steps one by one over from drone/travis/jenkins/whatever to travis/jenkins/drone/whatever, losing interest half way down the line and still have a stupid mess on our hands. It is stupid to use 5 different CI systems in one repo and I'm working towards reducing that down to one or two.