[#46926] - [5.4] Fix system test artifact name collision in CI matrix job
- Pending
- Medium
- Build: 5.4-dev
- # 46926
- Diff
- D3S-Gaurav:fix/ci-system-test-artifact-name-collision
User tests: Successful: Unsuccessful:
Pull Request resolves #46925.
- I read the Generative AI policy and my contribution is either not created with the help of AI or is compatible with the policy and GNU/GPL 2 or later.
Summary of Changes
The tests-system job in ci.yml runs in a matrix with 12 combinations (6 configs × 2 browsers), but the upload-artifact step uses a static artifact name system-test-output.
With actions/upload-artifact@v4, duplicate artifact names cause upload failures (unlike v3, which merged them). Only the first completed matrix run's artifact gets saved; the rest fail.
Changes made:
- Made the artifact name unique per matrix combination:
system-test-output-${{ matrix.browser }}-${{ matrix.config.test_group }} - Fixed a minor typo in the step name: "Archive test results results" → "Archive test results"
Testing Instructions
- Open .github/workflows/ci.yml, line 358-366
- Verify the step name reads
Archive test results(not "results results") - Verify the artifact name is
system-test-output-${{ matrix.browser }}-${{ matrix.config.test_group }} - Trigger a CI run and confirm all 12 matrix combinations upload artifacts without conflicts
Actual result BEFORE applying this Pull Request
All 12 matrix jobs in tests-system attempt to upload an artifact with the same static name system-test-output. With upload-artifact@v4, this causes upload failures for all but the first job to complete.
Expected result AFTER applying this Pull Request
Each matrix combination uploads its artifact with a unique name (e.g., system-test-output-chrome-cmysqlmax, system-test-output-edge-cpostgres), preventing upload conflicts.
Link to documentations
Please select:
-
Documentation link for guide.joomla.org:
-
No documentation changes for guide.joomla.org needed
-
Pull Request link for manual.joomla.org:
-
No documentation changes for manual.joomla.org needed
| Status | New | ⇒ | Pending |
joomla-cms-bot
- | Category | ⇒ | Repository |
| Labels |
Added:
bug
PR-5.4-dev
|
||
@D3S-Gaurav When GitHub shows that the branch if not up to date, it is not a problem as long as there are no conflicts shown. So it does not need to update the branch every time. Updating the branch invalidates our human test count (which is not the case for this PR here as there were no human tests, but it might be the case for other PRs). That's why it should only be done when necessary (i.e. conflicts to be resolved) or by a maintainer who can restore the human test count after that.
@tecpromotion The „no files found“ at the end is normal? We only have screenshots when something goes wrong?
I just see in logs of other PRs that it is normal as we only have screenshots if some test failed.
Hi @tecpromotion ,
Is there an estimated timeline for merging this PR? Just wanted to plan accordingly.
Thank you!
Hi @richard67,
You're right that usually only one job fails and the rest get cancelled.
However, there are still a couple of sneaky situations where this PR is helpful:
- Simultaneous failures: If two tests happen to fail at the exact same moment across different jobs, both will try to upload screenshots before the cancellation kicks in. The second upload will crash upload-artifact@v4 entirely because of the duplicate name.
- Easier debugging: Getting an artifact named system-test-output-chrome-cmysqlmax makes it instantly obvious which environment failed without having to dig through the logs.
- Testing all environments: Sometimes we might want to temporarily disable the "cancel everything on failure" behavior so we can see all the environments a new PR breaks. Without unique names, only the very first failure gets saved and the rest are lost.
![avatar github-actions[bot]](/images/avatars/github-actions[bot].png)
This pull request has conflicts, please resolve those before we can evaluate the pull request.
@tecpromotion The „no files found“ at the end is normal? We only have screenshots when something goes wrong?
A real test for working uploads could be to make a change in the system tests so they fail with one commit in that PR and then revert that commit with another commit. The logs for the failed commit should then show the successful uploads.