Problem

Moved from joomla-projects/joomla-pythagoras#73

The Automated Testing Working Group decided to use CodeCeption for testing. In order to provide consistency, the test environment of the CMS should be moved to CodeCeption.

Although CodeCeption supports PHPUnit tests, and actually uses PHPUnit to execute these tests, the test results are not identical for the two environments, as more tests are failing or even throwing errors, when executed with CodeCeption. It has shown to be a huge efford to rewrite the existing tests. The tests are not worth the efford, though, because of at least two reasons:

• a lot of tests use Reflection to test implementation details instead of API behaviour
• nearly all tests test things not used/needed in Joomla!4

Solution

The issue can be solved by making the transition stepwise.

Immediate Steps (4.0)

1. Move current tests directory to tests.phpunit and change phpunit.xml.dist, travisci-phpunit.xml, and other related files accordingly, so the current build procedure works with PHPUnit using the new directory. Done.
2. Add the CodeCeption test structure, with additional cli and api suites, in the tests directory. Done.
3. Add CodeCeption test environment to Travis configuration.
4. Write new tests for CodeCeption as needed. If code changes require changing PHPUnit tests, move these tests to the CodeCeption environment, rewriting them to test API instead of implementation, if needed.

Future Steps (4.x)

1. Provide PageObjects with a template-aware factory, so the same tests can be executed with any template.
2. Provide Docker and / or Vagrant support for integration (functional) and system (acceptance) tests.
3. Move PageObject, Docker, and Vagrant support to an external library, so they can be reused by third party developers.