No Code Attached Yet
avatar brianteeman
brianteeman
11 Feb 2025

The system tests for tuf_metadata are completely wrong. I dont really know enough about the tests but after they have run the #tuf_metadata table is empty

db_setValidTufRoot deletes any existing tuf_metadata instead of creating metadata

it looks like its just a copy paste of db_setInvalidTufRoot

Cypress.Commands.add('db_setValidTufRoot', () => {
cy.task('queryDB', 'DELETE FROM #__tuf_metadata WHERE id = 1');
cy.task('queryDB', 'DELETE FROM #__updates WHERE update_site_id = 1');
cy.task('queryDB', createInsertQuery(
'tuf_metadata',
{
id: 1,
update_site_id: 1,
root: JSON.stringify(validTufMetadata.root),
targets: '',
snapshot: '',
timestamp: '',
},

avatar brianteeman brianteeman - open - 11 Feb 2025
avatar joomla-cms-bot joomla-cms-bot - change - 11 Feb 2025
Labels Added: No Code Attached Yet
avatar joomla-cms-bot joomla-cms-bot - labeled - 11 Feb 2025
avatar brianteeman
brianteeman - comment - 11 Feb 2025

cc @SniperSister as it was your PR #42799 that introduced this faulty test

avatar brianteeman brianteeman - change - 11 Feb 2025
The description was changed
avatar brianteeman brianteeman - edited - 11 Feb 2025
avatar SniperSister
SniperSister - comment - 11 Feb 2025

Can't follow you. The test indeed removes any previous metadata and inserts either valid or invalid root metadata in a new row.

avatar brianteeman
brianteeman - comment - 11 Feb 2025

did you try it?

Image

avatar SniperSister
SniperSister - comment - 11 Feb 2025

Did I inspect the DB content during a test? Nope. However, the corresponding system tests check for different system messages that depend on the existence of the respective rows in the database. An empty metadata table would create a different error - that's why I'm assuming that the tests work as expected in our testing setup:

it('Can fetch available updates with valid metadata', () => {

avatar brianteeman
brianteeman - comment - 11 Feb 2025

after running the tests the db is empty

avatar heelc29
heelc29 - comment - 11 Feb 2025

Please check #44861.

avatar SniperSister
SniperSister - comment - 11 Feb 2025

So, maybe I'm missing something here: what's the usecase of those system tests? So far I was under the assumption that they would be run "from scratch", creating a new installation with sample content for each and every run; if that's the case, where's the problem with the empty TUF table in the first place?

And if it's really an issue, wouldn't it be the proper fix to restore the default, valid root data after each test?

avatar brianteeman
brianteeman - comment - 11 Feb 2025

The tests should be repeatable. As far as I can assert they are not because the command to create valid tufdata which should run after each individual test doesn't do that.

avatar SniperSister SniperSister - change - 12 Feb 2025
Status New Closed
Closed_Date 0000-00-00 00:00:00 2025-02-12 07:14:43
Closed_By SniperSister
avatar SniperSister SniperSister - close - 12 Feb 2025
avatar laoneo
laoneo - comment - 12 Feb 2025

For performance reasons we do not do a new installation before every test, but a cleanup afterwards (actually it would be better to prepare the system in a state before the test runs in the actual spec file). But we decided for a cleanup afterwards (on some point we probably change that, but for now it is as it is). So I do agree here that in after each, we should leave the database in the state as it was before the test run.

Add a Comment

Login with GitHub to post a comment