Joomla! Issue Tracker | Joomla! CMS #42799 - [5.1] TUF-based core updates

Feature RTC Unit/System Tests Language Change Composer Dependency Changed Release Blocker PR-5.1-dev

Fixed in Code Base
26 Feb 2024
Medium
Build: 5.1-dev
# 42799
Diff
SniperSister:snipersister/tuf-j5

Pending

User tests: Successful: Unsuccessful:

SniperSister
12 Feb 2024

Summary of Changes

This pull requests changes the way how Joomla retrieves update information for Joomla core.

So far, the information about available updates has been retrieved using an XML file hosted on the Joomla.org CDN. Whatever information was written in that XML file was trusted and there was no way for a Joomla installation of that update XML actually is a legit file distributed by the project.

This makes the project vulnerable to supply chain attacks, where an attacker, once he gains access to the update XML file, might be able to distribute malicious update packages. The already implemented security measure of package hashes is no proper mitigation for that scenario as the package URL und the package hashes are stored in the same XML.

In order to succesfully mitigate such attacks, we would like to use "The Update Framework" (short "TUF") to the Joomla core updater. We are not going to introduce the general concepts of TUF in this PR as it's very extensively documented at https://theupdateframework.io/

The main changes in this PR are:

Inclusion of the PHP-TUF client
Inclusion of new library classes to connect the TUF client with the CMS
Addition of a new service provider for the HTTP Factory, which allows us to mock it as a dependency in our unit tests
Various changes to the existing Update and Updater classes to add TUF repos as a potential update source next to the existing XML mechanism
An additional check in com_joomlaupdate to verify that the package version that shall be installed is actually the package version that users confirmed to install - that fixes an existing bug, where the re-retrieval of update information before the package download might cause a different version to be installed than the version that user saw on the update information page

Testing Instructions

Preparation steps

Apply the patch
Update the composer dependencies with composer install
Execute the DB changes by navigating to System > Maintenance > Database and hit "Update Structure"

Scenario 1: successful retrieval of a legit core update via TUF

Execute the preparation steps above if not done yet
Execute the "valid production metadata" query from the test queries section below using a DB client of your choice
Navigate to System > Update > Joomla
Fetch the available updates by clicking "Check for Updates" in the toolbar
Expected Result: Fetching succeeds, Message "Checked for updates.", no update being offered

Scenario 2: blocked retrieval of a malicious core update via TUF

Execute the preparation steps above if not done yet
Execute the "invalid test metadata" query from the test queries section below using a DB client of your choice
Navigate to System > Update > Joomla
Fetch the available updates by clicking "Check for Updates" in the toolbar
Expected Result: The message "Update not possible because the offered update does not have enough signatures" is shown.

Scenario 3: successful retrieval of a core update via a custom XML server

Execute the preparation steps above if not done yet
Navigate to System > Update > Joomla, hit "Options"
Set the Update Channel to "Custom" and use https://update.joomla.org/core/sts/list_sts.xml as an update URL
Hit save & close
Fetch the available updates by clicking "Check for Updates" in the toolbar
Expected Result: Fetching succeeds, Message "Checked for updates.", no update being offered
Switch the Update Channel back to Default

Scenario 4: successful retrieval of an extension update via the existing XML mechanisms

Execute the preparation steps above if not done yet
Execute the "valid production metadata" query from the test queries section below using a DB client of your choice
Install an outdated version of an extension of choice that supports the Joomla updater
Fetch and install the extension update
Expected result: Fetching succeeds, update can be installed

Scenario 5: reinstall feature is available

Execute the preparation steps above if not done yet
Execute the "valid production metadata" query from the test queries section below using a DB client of your choice
Open the file administrator/components/com_joomlaupdate/src/Model/UpdateModel.php and change line 119 from $updateURL = 'https://update.joomla.org/cms/'; to $updateURL = 'https://update.joomla.org/alpha/';
Navigate to System > Update > Joomla
Fetch the available updates by clicking "Check for Updates" in the toolbar
Fetching succeeds, update to 5.1.100 being offered
Modify the stored core update in the #__updates table and change the version from 5.1.100 to 5.1.0-alpha4-dev (or whatever your local 5.1.x version is)
Expected: A screen that allows to reinstall the core files is shown:

Scenario 6: Constraint information is availabel

Execute the preparation steps above if not done yet
Execute the "valid production metadata" query from the test queries section below using a DB client of your choice
Open the file administrator/components/com_joomlaupdate/src/Model/UpdateModel.php and change line 119 from $updateURL = 'https://update.joomla.org/cms/'; to $updateURL = 'https://update.joomla.org/alpha/';
Navigate to System > Update > Joomla
Fetch the available updates by clicking "Check for Updates" in the toolbar
Fetching succeeds, update to 5.1.100 being offered
Modify the file libraries/src/Updater/ConstraintChecker.php, line 151, replace if (!$result) { with if (!$result || true) { to simulate a failed php constraint
Expected: A screen with information about the failed constraint is shown

Scenario 7: successful installation of a core update

Execute the preparation steps above if not done yet
Execute the "valid production metadata" query from the test queries section below using a DB client of your choice
Open the file administrator/components/com_joomlaupdate/src/Model/UpdateModel.php and change line 119 from $updateURL = 'https://update.joomla.org/cms/'; to $updateURL = 'https://update.joomla.org/alpha/';
Navigate to System > Update > Joomla
Fetch the available updates by clicking "Check for Updates" in the toolbar
Fetching succeeds, update to 5.1.100 being offered
Expected result: Update to 5.1.100 can be performed

Test Queries

Valid production metadata - MySQL and MariaDB

DELETE FROM `#__tuf_metadata`;

INSERT INTO `#__tuf_metadata` (`id`, `update_site_id`, `root`, `targets`, `snapshot`, `timestamp`, `mirrors`) VALUES
(1, 1, '{"signed":{"_type":"root","spec_version":"1.0","version":4,"expires":"2025-03-02T16:38:55Z","keys":{"07eb082f367c034a95878687f6648aa76d93652b6ee73e58817053d89af6c44f":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"9b2af2d9b9727227735253d795bd27ea8f0e294a5f3603e822dc5052b44802b9"}},"1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"a18e5ebabc19d5d5984b601a292ece61ba3662ab2d071dc520da5bd4f8948799"}},"2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"cb0a7a131961a20edea051d6dc2b091fb650bd399bd8514adb67b3c60db9f8f9"}},"31dd7c7290d664c9b88c0dead2697175293ea7df81b7f24153a37370fd3901c3":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"589d029a68b470deff1ca16dbf3eea6b5b3fcba0ae7bb52c468abc7fb058b2a2"}},"9e41a9d62d94c6a1c8a304f62c5bd72d84a9f286f27e8327cedeacb09e5156cc":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"6043c8bacc76ac5c9750f45454dd865c6ca1fc57d69e14cc192cfd420f6a66a9"}},"e2229942b0fc1e6d7f82adf258e5bdadac10046d1470b7ec459c9eb4e076026b":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"ad1950e117b29ebe7a38635a2e574123e07571e4f9a011783e053b5f15d2562a"}},"ecc851a051c8d6439331ff0a37c7727321fc39896a34f950f73638b8a7cb472e":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"5d451915bc2b93a0e4e4745bc6a8b292d58996d50e0fb66c78c7827152a65879"}}},"roles":{"root":{"keyids":["1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669","2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e"],"threshold":1},"snapshot":{"keyids":["07eb082f367c034a95878687f6648aa76d93652b6ee73e58817053d89af6c44f","2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e","ecc851a051c8d6439331ff0a37c7727321fc39896a34f950f73638b8a7cb472e","e2229942b0fc1e6d7f82adf258e5bdadac10046d1470b7ec459c9eb4e076026b"],"threshold":1},"targets":{"keyids":["31dd7c7290d664c9b88c0dead2697175293ea7df81b7f24153a37370fd3901c3","ecc851a051c8d6439331ff0a37c7727321fc39896a34f950f73638b8a7cb472e","e2229942b0fc1e6d7f82adf258e5bdadac10046d1470b7ec459c9eb4e076026b"],"threshold":1},"timestamp":{"keyids":["9e41a9d62d94c6a1c8a304f62c5bd72d84a9f286f27e8327cedeacb09e5156cc"],"threshold":1}},"consistent_snapshot":true},"signatures":[{"keyid":"1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669","sig":"1c8060aab4c5290dc398199d8f124701bd3f7d3fb47d688e3e61d20eeb90d6e387556ce680ba8db9b99f15332df64da349a03344f50ab4f1fe491efdf88f170c"}]}', NULL, NULL, NULL, NULL);

Valid production metadata - PostgreSQL

DELETE FROM "#__tuf_metadata";

INSERT INTO "#__tuf_metadata" ("id", "update_site_id", "root", "targets", "snapshot", "timestamp", "mirrors") VALUES
(1, 1, '{"signed":{"_type":"root","spec_version":"1.0","version":4,"expires":"2025-03-02T16:38:55Z","keys":{"07eb082f367c034a95878687f6648aa76d93652b6ee73e58817053d89af6c44f":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"9b2af2d9b9727227735253d795bd27ea8f0e294a5f3603e822dc5052b44802b9"}},"1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"a18e5ebabc19d5d5984b601a292ece61ba3662ab2d071dc520da5bd4f8948799"}},"2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"cb0a7a131961a20edea051d6dc2b091fb650bd399bd8514adb67b3c60db9f8f9"}},"31dd7c7290d664c9b88c0dead2697175293ea7df81b7f24153a37370fd3901c3":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"589d029a68b470deff1ca16dbf3eea6b5b3fcba0ae7bb52c468abc7fb058b2a2"}},"9e41a9d62d94c6a1c8a304f62c5bd72d84a9f286f27e8327cedeacb09e5156cc":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"6043c8bacc76ac5c9750f45454dd865c6ca1fc57d69e14cc192cfd420f6a66a9"}},"e2229942b0fc1e6d7f82adf258e5bdadac10046d1470b7ec459c9eb4e076026b":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"ad1950e117b29ebe7a38635a2e574123e07571e4f9a011783e053b5f15d2562a"}},"ecc851a051c8d6439331ff0a37c7727321fc39896a34f950f73638b8a7cb472e":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"5d451915bc2b93a0e4e4745bc6a8b292d58996d50e0fb66c78c7827152a65879"}}},"roles":{"root":{"keyids":["1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669","2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e"],"threshold":1},"snapshot":{"keyids":["07eb082f367c034a95878687f6648aa76d93652b6ee73e58817053d89af6c44f","2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e","ecc851a051c8d6439331ff0a37c7727321fc39896a34f950f73638b8a7cb472e","e2229942b0fc1e6d7f82adf258e5bdadac10046d1470b7ec459c9eb4e076026b"],"threshold":1},"targets":{"keyids":["31dd7c7290d664c9b88c0dead2697175293ea7df81b7f24153a37370fd3901c3","ecc851a051c8d6439331ff0a37c7727321fc39896a34f950f73638b8a7cb472e","e2229942b0fc1e6d7f82adf258e5bdadac10046d1470b7ec459c9eb4e076026b"],"threshold":1},"timestamp":{"keyids":["9e41a9d62d94c6a1c8a304f62c5bd72d84a9f286f27e8327cedeacb09e5156cc"],"threshold":1}},"consistent_snapshot":true},"signatures":[{"keyid":"1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669","sig":"1c8060aab4c5290dc398199d8f124701bd3f7d3fb47d688e3e61d20eeb90d6e387556ce680ba8db9b99f15332df64da349a03344f50ab4f1fe491efdf88f170c"}]}', NULL, NULL, NULL, NULL);

Invalid test metadata - MySQL and MariaDB

DELETE FROM `#__tuf_metadata`;

INSERT INTO `#__tuf_metadata` (`id`, `update_site_id`, `root`, `targets`, `snapshot`, `timestamp`, `mirrors`) VALUES
(1, 1, '{"signed":{"_type":"root","spec_version":"1.0","version":1,"expires":"2028-12-06T15:31:52Z","keys":{"1689c5951cfc8a8cb4e3535c6ddc3f8d5c66e2effd4b7aae3506995f145da2a0":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"71c24873013b6f21aca791f45dcd9ddb5842a97bf72ac73c211742c2659a97ff"}},"696a7598c714e545bb8a3a4248d82bf4c66486d142e226c1e06601a14f4d939a":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"9fac963aac4e14f948a7c2d6b3fa2232f6cb5a08bf6a8b6100bc6e68b0683c1c"}},"70c4fb4ffe87b8d75559092c75bc038d587790bf2ecb9d8d6c6c0fae6705c750":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"d08225342af7a8075bf210bd62154567140a8e14d824743e58b8e7e64ee8ad0b"}},"92933ea840e57ad3db67c748d1a309c4a7d8be3f70d8bbbd3cff9c4cca3bcf7b":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"8d70ac7574e64f209bff3d7c1d8b8ab6e34cf4419dd09f0d222354dceee986d7"}},"f9854d7c61e9413f4d83678be7d50310cc9e062027746d8936ba4736e75224e9":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"b7a3d08989b5885d78e93425daacf3a71b0e190759e1a8633aa41bdb3ec3cd97"}}},"roles":{"root":{"keyids":["70c4fb4ffe87b8d75559092c75bc038d587790bf2ecb9d8d6c6c0fae6705c750"],"threshold":1},"snapshot":{"keyids":["f9854d7c61e9413f4d83678be7d50310cc9e062027746d8936ba4736e75224e9"],"threshold":1},"targets":{"keyids":["696a7598c714e545bb8a3a4248d82bf4c66486d142e226c1e06601a14f4d939a"],"threshold":1},"timestamp":{"keyids":["1689c5951cfc8a8cb4e3535c6ddc3f8d5c66e2effd4b7aae3506995f145da2a0","92933ea840e57ad3db67c748d1a309c4a7d8be3f70d8bbbd3cff9c4cca3bcf7b"],"threshold":1}},"consistent_snapshot":true},"signatures":[{"keyid":"70c4fb4ffe87b8d75559092c75bc038d587790bf2ecb9d8d6c6c0fae6705c750","sig":"52f8de5d8c0ac8c532a4e3c274b3e22cd2dca57a9f5d4094ccc1ded9966fb7064acc589ad564ba7ba04f7dfb42d8ccb803811b73551c60df4f9996c116967e00"}]}', NULL, NULL, NULL, NULL);

Invalid test metadata - PostgreSQL

DELETE FROM "#__tuf_metadata";

INSERT INTO "#__tuf_metadata" ("id", "update_site_id", "root", "targets", "snapshot", "timestamp", "mirrors") VALUES
(1, 1, '{"signed":{"_type":"root","spec_version":"1.0","version":1,"expires":"2028-12-06T15:31:52Z","keys":{"1689c5951cfc8a8cb4e3535c6ddc3f8d5c66e2effd4b7aae3506995f145da2a0":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"71c24873013b6f21aca791f45dcd9ddb5842a97bf72ac73c211742c2659a97ff"}},"696a7598c714e545bb8a3a4248d82bf4c66486d142e226c1e06601a14f4d939a":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"9fac963aac4e14f948a7c2d6b3fa2232f6cb5a08bf6a8b6100bc6e68b0683c1c"}},"70c4fb4ffe87b8d75559092c75bc038d587790bf2ecb9d8d6c6c0fae6705c750":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"d08225342af7a8075bf210bd62154567140a8e14d824743e58b8e7e64ee8ad0b"}},"92933ea840e57ad3db67c748d1a309c4a7d8be3f70d8bbbd3cff9c4cca3bcf7b":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"8d70ac7574e64f209bff3d7c1d8b8ab6e34cf4419dd09f0d222354dceee986d7"}},"f9854d7c61e9413f4d83678be7d50310cc9e062027746d8936ba4736e75224e9":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"b7a3d08989b5885d78e93425daacf3a71b0e190759e1a8633aa41bdb3ec3cd97"}}},"roles":{"root":{"keyids":["70c4fb4ffe87b8d75559092c75bc038d587790bf2ecb9d8d6c6c0fae6705c750"],"threshold":1},"snapshot":{"keyids":["f9854d7c61e9413f4d83678be7d50310cc9e062027746d8936ba4736e75224e9"],"threshold":1},"targets":{"keyids":["696a7598c714e545bb8a3a4248d82bf4c66486d142e226c1e06601a14f4d939a"],"threshold":1},"timestamp":{"keyids":["1689c5951cfc8a8cb4e3535c6ddc3f8d5c66e2effd4b7aae3506995f145da2a0","92933ea840e57ad3db67c748d1a309c4a7d8be3f70d8bbbd3cff9c4cca3bcf7b"],"threshold":1}},"consistent_snapshot":true},"signatures":[{"keyid":"70c4fb4ffe87b8d75559092c75bc038d587790bf2ecb9d8d6c6c0fae6705c750","sig":"52f8de5d8c0ac8c532a4e3c274b3e22cd2dca57a9f5d4094ccc1ded9966fb7064acc589ad564ba7ba04f7dfb42d8ccb803811b73551c60df4f9996c116967e00"}]}', NULL, NULL, NULL, NULL);

Link to documentations

Please select:

No documentation changes for docs.joomla.org needed
No documentation changes for manual.joomla.org needed
Link to internal documentation: https://internal.joomla.org/docs/production/update/infrastructure/expired-metadata

Kudos

This is not my personal work, a ton of people helped creating this feature and I would like to thank Harald, Benjamin, Niels, Martina, Hannes, Magnus, Tobias, Franciska, Timo, Stefan and Elias for their time and contributions!

f765340 20 Mar 2022

Set up tuf_updates mysql queries

fd82ae8 20 Mar 2022

Fix table structure for TUF metadatas

f43a95e 20 Mar 2022

Change ordering of timestamp

68ec0b1 20 Mar 2022

Add allow null

37d2341 20 Mar 2022

Fix null values

5909617 20 Mar 2022

Merge pull request #2 from bembelimen/tuf/db-structure

351bb65 20 Mar 2022

Add TUF databaseStorage

8e5bffb 20 Mar 2022

Merge branch 'tuf-client' of github.com:fancyFranci/joomla-cms into tuf-client

e69ec81 20 Mar 2022

Create tuf table

7571ead 20 Mar 2022

Add TUF exceptions

702d212 20 Mar 2022

Implement all ArrayAccess methods

69055f3 20 Mar 2022

Merge pull request #3 from bembelimen/tuf/databasestorage

e83a573 20 Mar 2022

Set TUF exception

798bc72 23 Mar 2022

implement basic version of the tuf validator

0a3f0b4 23 Mar 2022

remove php-tuf library

b06ed49 10 Jun 2022

What zero-24 says

5b8e736 10 Jun 2022

Merge branch '4.2-dev' into tuf-client

130e62d 10 Jun 2022

Merge branch '4.2-dev' into validator

9b62a03 10 Jun 2022

Merge remote-tracking branch 'magnus-tuf/validator' into tuf-client

133eacd 10 Jun 2022

Merge pull request #3 from fancyFranci/tuf-client

f81e015 10 Jun 2022

Update drone hash

7934f52 10 Jun 2022

Add symfony/validator into composer.lock

7fd25c8 10 Jun 2022

Codestyle fixes

dd22ede 10 Jun 2022

Fix code style

93fcbfb 11 Jun 2022

wip tuf

effde12 11 Jun 2022

Change GuzzleFilteFetcher to HttpFileFetcher

3b057a8 11 Jun 2022

add missing semicolon

1da619b 11 Jun 2022

initial wip

f6ac9c5 11 Jun 2022

doc block updates

5585d32 11 Jun 2022

wip php minimum & stability

SniperSister - open - 12 Feb 2024

SniperSister - change - 12 Feb 2024

Status

New

⇒

Pending

joomla-cms-bot - change - 12 Feb 2024

Scenario 1

Expected

Currently no update available.

Actual

Check if an update is available.

Scenario 2

Expected

Update not possible because the offered update does not have enough signatures

Actual

Check if an update is available.

Scenario 3

Expected

Currently no update available.

Actual

Check if an update is available.

brianteeman - test_item - 12 Feb 2024 - Tested unsuccessfully

brianteeman - comment - 12 Feb 2024

I have tested this item ? unsuccessfully on 5585d32

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

alikon - comment - 12 Feb 2024

anyway not a pr for a minor release if we still follow SEMver

joomla-cms-bot - change - 13 Feb 2024

Scenario 1

Before PR

With PR

Comment

Losing the text change is not a good change and losing the ability to reinstall

brianteeman - comment - 13 Feb 2024

Scenario 2

PHP Version 8.1.10
Web Server Apache/2.4.54 (Win64) OpenSSL/1.1.1q PHP/8.1.10

SniperSister - change - 13 Feb 2024

Labels

Removed: NPM Resource Changed

SniperSister - comment - 13 Feb 2024

@brianteeman the notice in scenario 2 has been fixed.

Regarding scenario 1: I'm unable to reproduce the original screen in a vanilla 5.1-dev environment. Any hints? What updateserver is configured?

richard67 - comment - 13 Feb 2024

@SniperSister What will be shown when there is a valid update available but the server doesn’t meet the target version’s requirements for the PHP or database version? Does it show that, or does it just show up hat no update was found? I had made PR #42489 for that.

brianteeman - comment - 13 Feb 2024

Regarding scenario 1: I'm unable to reproduce the original screen in a vanilla 5.1-dev environment. Any hints? What updateserver is configured?

This is the basic core functionality as created by @bembelimen here #34754

Without it we lose the ablity to reupload the same version and to have useful messages

SniperSister - comment - 13 Feb 2024

This is the basic core functionality as created by @bembelimen here #34754

I'm not questioning that this is useful, I'm just wondering how you are able to reproduce the screen with a vanialla 5.1-dev installation as the current update servers don't offer an update matching the current dev branch and therefore the screen in question can't be generated.

brianteeman - comment - 13 Feb 2024

~~Not doing anything special. Clean checkout of 5.1 branch. Its the same behaviour on a live install of 5.0.2~~

No ideas as I dont see it now

SniperSister - comment - 13 Feb 2024

No ideas as I dont see it now

Ok, that matches my result! Will look into it with @bembelimen

SniperSister - change - 17 Feb 2024

The description was changed

SniperSister - edited - 17 Feb 2024

SniperSister - change - 17 Feb 2024

The description was changed

SniperSister - edited - 17 Feb 2024

SniperSister - comment - 17 Feb 2024

@richard67 @brianteeman I've added test scenarios addressing the two cases that you have mentioned.

HLeithner - comment - 18 Feb 2024

Checking for constrains and providing to the user works now.

Showing update but local failure works too:

After repairing the database the update shows the ready for update screen

web54 - comment - 24 Feb 2024

scenario 1 : when patch applied : error : The file marked for modification does not exist: composer.json
i can not patch

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

richard67 - comment - 24 Feb 2024

scenario 1 : when patch applied : error : The file marked for modification does not exist: composer.json i can not patch

@web54 The PR has a conflict which needs to be resolved first. In addition, it might be that it's not possible to test it with the Patchtester component. It might need to use the update packages or custom update URL provided with the Download link at the bottom of this PR. But please wait with testing until GitHub doesn't show the conflicting files anymore at the bottom of the PR. Anyway thanks for your testing attempt.

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

Labels

Added: Release Blocker

richard67 - comment - 24 Feb 2024

@SniperSister Shall I resolve the conflicts or will you do?

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

richard67 - comment - 24 Feb 2024

@SniperSister I think you've resolved the conflict in the model in the wrong way. You have reverted the changes from PR #42603 where the code comments for cases of switch statements were moved and where a default case was added.

Update: False alarm, all ok.

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

SniperSister - change - 24 Feb 2024

The description was changed

SniperSister - edited - 24 Feb 2024

Elfangor93 - test_item - 24 Feb 2024 - Tested successfully

Elfangor93 - comment - 24 Feb 2024

I have tested this item ✅ successfully on 5585d32

Thank you very much for your great effort!
Tested on a Windows machine with php 8.1.13.

All described scenarios were successfully tested. After scenario 7 when the update was fully run through I get a php error stating that Resource 'Joomla\CMS\Http\HttpFactoryInterface' has not been registered with the container. But this shouldnt be caused by this PR. So I mark this as successful.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

SniperSister - comment - 24 Feb 2024

After scenario 7 when the update was fully run through I get a php error stating that Resource 'Joomla\CMS\Http\HttpFactoryInterface' has not been registered with the container.

That's indeed not related to the actual PR but to the fact that the 5.1.100 release is an older 5.1.x package which causes incompatibilites AFTER the update has been applied successfully.

richard67 - comment - 24 Feb 2024

@SniperSister It needs to rename the update SQL scripts from "5.1.0-2023-12-09.sql" to something newer than "5.1.0-2024-02-10.sql" as that is the newest one in the current 5.1-dev branch. Otherwise your scripts will not run then updating from 5.1.0-alpha4 to beta1. I suggest to use "5.1.0-2024-02-24.sql".

degobbis - test_item - 24 Feb 2024 - Tested successfully

degobbis - comment - 24 Feb 2024

I have tested this item ✅ successfully on 5585d32

Tested all 7 scenarios successfully with php 8.2 and MariaDB 10.11

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

SniperSister - comment - 24 Feb 2024

Done @richard67

richard67 - alter_testresult - 24 Feb 2024 - brianteeman: Not tested

richard67 - alter_testresult - 24 Feb 2024 - degobbis: Tested successfully

richard67 - alter_testresult - 24 Feb 2024 - Elfangor93: Tested successfully

richard67 - comment - 24 Feb 2024

After the renaming of the update SQL scripts I've restored the previous test results as that change did not have any impact on the test result. But it would be good to get one test in addition which tests updating from 5.1.0-alpha4 or any older version down to 4.4.3 just to make sure we don't have any mistake in the update SQL.

joomdonation - comment - 24 Feb 2024

Tested all the scenarios works as described but in PHP error logs, there are error messages below (I'm unsure if this is related to the change from this PR)

[24-Feb-2024 14:17:14 UTC] PHP Deprecated: Creation of dynamic property Joomla\CMS\Updater\Update::$jversion.full is deprecated in [ROOT]\libraries\src\Object\LegacyPropertyManagementTrait.php on line 135

[24-Feb-2024 14:17:15 UTC] PHP Deprecated: Creation of dynamic property Joomla\CMS\Updater\Update::$folder is deprecated in [ROOT]\libraries\src\Updater\Update.php on line 475

SniperSister - comment - 24 Feb 2024

@joomdonation both notices are unrelated to this PR

nadjak77 - test_item - 24 Feb 2024 - Tested successfully

nadjak77 - comment - 24 Feb 2024

I have tested this item ✅ successfully on 5585d32

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

SniperSister - comment - 24 Feb 2024

But it would be good to get one test in addition which tests updating from 5.1.0-alpha4 or any older version down to 4.4.3 just to make sure we don't have any mistake in the update SQL.

I don't get that request. What do you want us to do? Executing the update statements in an older Joomla version?

richard67 - comment - 24 Feb 2024

But it would be good to get one test in addition which tests updating from 5.1.0-alpha4 or any older version down to 4.4.3 just to make sure we don't have any mistake in the update SQL.

I don't get that request. What do you want us to do? Executing the update statements in an older Joomla version?

I want that the update SQL are run, and this happens when you update with the update component.

SniperSister - comment - 24 Feb 2024

I want that the update SQL are run, and this happens when you update with the update component.

Ok, and how can we simulate that? The feature is not released, so there's no package that once could update too?

richard67 - comment - 24 Feb 2024

I want that the update SQL are run, and this happens when you update with the update component.

Ok, and how can we simulate that? The feature is not released, so there's no package that once could update too?

Update any version equal to or older than 5.1.0-alpha4 to the package created by drone for this PR here or use the update URL created by drone, that's what I mean.

SniperSister - comment - 24 Feb 2024

Ok got it :)

SniperSister - comment - 24 Feb 2024

@richard67 applied the update package on a 5.1-dev release, update could be applied successfully, table and table content is created as expected:

richard67 - alter_testresult - 24 Feb 2024 - Elfangor93: Tested successfully

richard67 - alter_testresult - 24 Feb 2024 - degobbis: Tested successfully

richard67 - alter_testresult - 24 Feb 2024 - nadjak77: Tested successfully

richard67 - comment - 24 Feb 2024

I've restored the previous human test results in the issue tracker as the commit which invalidated the count did not change anything functional.

richard67 - test_item - 24 Feb 2024 - Tested unsuccessfully

richard67 - comment - 24 Feb 2024

I have tested this item ? unsuccessfully on 5585d32

Updating from 4.4-dev to the package created by drone for this PR fails on PostgreSQL with an SQL error.

2024-02-24 16:52:10.791 CET [26538] jdb1adm@joomladb1 ERROR:  column "file" does not exist at character 226
2024-02-24 16:52:10.791 CET [26538] jdb1adm@joomladb1 STATEMENT:  INSERT INTO "j3ux0_tuf_metadata" ("update_site_id", "root")
	VALUES ((SELECT ue."update_site_id" FROM "j3ux0_update_sites_extensions" AS ue JOIN "j3ux0_extensions" AS e ON (e."extension_id" = ue."extension_id") WHERE e."type"="file" AND e."element"='joomla'), '{"signed":{"_type":"root","spec_version":"1.0","version":2,"expires":"2025-03-02T11:22:17Z","keys":{"07eb082f367c034a95878687f6648aa76d93652b6ee73e58817053d89af6c44f":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"9b2af2d9b9727227735253d795bd27ea8f0e294a5f3603e822dc5052b44802b9"}},"1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"a18e5ebabc19d5d5984b601a292ece61ba3662ab2d071dc520da5bd4f8948799"}},"2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"cb0a7a131961a20edea051d6dc2b091fb650bd399bd8514adb67b3c60db9f8f9"}},"31dd7c7290d664c9b88c0dead2697175293ea7df81b7f24153a37370fd3901c3":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"589d029a68b470deff1ca16dbf3eea6b5b3fcba0ae7bb52c468abc7fb058b2a2"}},"9e41a9d62d94c6a1c8a304f62c5bd72d84a9f286f27e8327cedeacb09e5156cc":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"6043c8bacc76ac5c9750f45454dd865c6ca1fc57d69e14cc192cfd420f6a66a9"}}},"roles":{"root":{"keyids":["1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669","2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e"],"threshold":1},"snapshot":{"keyids":["07eb082f367c034a95878687f6648aa76d93652b6ee73e58817053d89af6c44f","2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e"],"threshold":1},"targets":{"keyids":["31dd7c7290d664c9b88c0dead2697175293ea7df81b7f24153a37370fd3901c3"],"threshold":1},"timestamp":{"keyids":["9e41a9d62d94c6a1c8a304f62c5bd72d84a9f286f27e8327cedeacb09e5156cc"],"threshold":1}},"consistent_snapshot":true},"signatures":[{"keyid":"2dcaf3d0e552f150792f7c636d45429246dcfa34ac35b46a44f5c87cd17d457e","sig":"2a225a560ec0837b721d4c5e379fedbd3c7c9079a94e6b31e47e0184c8b95421b6036b4286c5d90f29ab4c468d79a712fdb65e96511394ceb3aa8e2b3983a501"},{"keyid":"1b1b1dd55b2c1c7258714cf1c1ae06f23e4607b28c762d016a9d81c48ffe5669","sig":"8ce0b2a7bdc1e6dcba12081f440510df0a593c072dcf591631c2dd0f456844a7da63be8e8ac31ffbddf42641fde84dc733a336031d182c2163b4c1eaf2117005"}]}');

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

richard67 - alter_testresult - 24 Feb 2024 - Elfangor93: Tested successfully

richard67 - alter_testresult - 24 Feb 2024 - degobbis: Tested successfully

richard67 - alter_testresult - 24 Feb 2024 - nadjak77: Tested successfully

richard67 - comment - 24 Feb 2024

I've restored again the previous human test results in the issue tracker as the commit which invalidated the count did not change anything functional on the TUF implementation, it only fixed updating from older versions with a PostgreSQL database, which was not subject of these tests anyway.

richard67 - alter_testresult - 24 Feb 2024 - Elfangor93: Tested successfully

richard67 - alter_testresult - 24 Feb 2024 - degobbis: Tested successfully

richard67 - alter_testresult - 24 Feb 2024 - nadjak77: Tested successfully

richard67 - comment - 24 Feb 2024

I've restored again the previous human test results in the issue tracker as the commit which invalidated the count was just a fix for code comments.

richard67 - test_item - 24 Feb 2024 - Tested unsuccessfully

richard67 - comment - 24 Feb 2024

I have tested this item ? unsuccessfully on 5585d32

Updating from 4.4.3 to the custom update URL created by drone for this PR fails. I will post details in the next comment.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

richard67 - comment - 24 Feb 2024

Updating from 4.4.3 to the custom update URL of this PR fails with the following error:

In PHP log:

[Sat Feb 24 18:52:00.749850 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP Warning:  Undefined property: stdClass::$location in /home/richard/lamp/public_html/test-1/administrator/components/com_joomlaupdate/src/Model/UpdateModel.php on line 327, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749923 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP Stack trace:, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749933 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   1. {main}() /home/richard/lamp/public_html/test-1/administrator/index.php:0, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749937 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   2. require_once() /home/richard/lamp/public_html/test-1/administrator/index.php:32, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749942 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   3. Joomla\\CMS\\Application\\CMSApplication->execute() /home/richard/lamp/public_html/test-1/administrator/includes/app.php:58, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749946 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   4. Joomla\\CMS\\Application\\AdministratorApplication->doExecute() /home/richard/lamp/public_html/test-1/libraries/src/Application/CMSApplication.php:306, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749964 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   5. Joomla\\CMS\\Application\\AdministratorApplication->dispatch($component = *uninitialized*) /home/richard/lamp/public_html/test-1/libraries/src/Application/AdministratorApplication.php:195, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749971 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   6. Joomla\\CMS\\Component\\ComponentHelper::renderComponent($option = 'com_joomlaupdate', $params = *uninitialized*) /home/richard/lamp/public_html/test-1/libraries/src/Application/AdministratorApplication.php:150, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749976 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   7. Joomla\\CMS\\Dispatcher\\ComponentDispatcher->dispatch() /home/richard/lamp/public_html/test-1/libraries/src/Component/ComponentHelper.php:361, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.749995 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   8. Joomla\\CMS\\MVC\\Controller\\BaseController->execute($task = 'ajax') /home/richard/lamp/public_html/test-1/libraries/src/Dispatcher/ComponentDispatcher.php:143, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.750000 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP   9. Joomla\\Component\\Joomlaupdate\\Administrator\\Controller\\UpdateController->ajax() /home/richard/lamp/public_html/test-1/libraries/src/MVC/Controller/BaseController.php:730, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/
[Sat Feb 24 18:52:00.750005 2024] [php:notice] [pid 30013] [client 192.168.98.1:51284] PHP  10. Joomla\\Component\\Joomlaupdate\\Administrator\\Model\\UpdateModel->getUpdateInformation() /home/richard/lamp/public_html/test-1/administrator/components/com_joomlaupdate/src/Controller/UpdateController.php:703, referer: https://www.test-1.vmubu01.vmnet2.local/administrator/

Updating from 4.4.x to any 5.y is a valid scenario and should work.

joomdonation - comment - 24 Feb 2024

And before I forgot, does anyone (maybe @richard67 ) know why in UpdateModel, we are using this kind of code to get database object:

$db    = version_compare(JVERSION, '4.2.0', 'lt') ? $this->getDbo() : $this->getDatabase();

Maybe it is not needed now? Asking this because the code in this PR always use $this->getDatabase(); without version check.

richard67 - comment - 24 Feb 2024

And before I forgot, does anyone (maybe @richard67 ) know why in UpdateModel, we are using this kind of code to get database object:
$db    = version_compare(JVERSION, '4.2.0', 'lt') ? $this->getDbo() : $this->getDatabase();
Maybe it is not needed now? Asking this because the code in this PR always use $this->getDatabase(); without version check.

I think this can indeed be simplified in 5.1-dev (or even 5.0-dev). We needed that in 4.4-dev when updating from versions older than 4.2.0.

richard67 - alter_testresult - 24 Feb 2024 - richard67: Not tested

richard67 - change - 24 Feb 2024

Build

4.4-dev

⇒

5.1-dev

richard67 - comment - 24 Feb 2024

On PostgreSQL the test queries cause a syntax error. When not escaping the double quotes in the JSON strings, they are working. Just wanted to mention so it is documented. The SQL in the base.sql and the update SQL scripts is ok as that doesn't escape the double quotes, so it is only the testing instructions which need a fix.

richard67 - comment - 24 Feb 2024

When testing scenario 4 5 on PostgreSQL I get an SQL error.

postgresql-14-main.log:

2024-02-24 20:35:13.973 CET [5632] jdb4adm@joomladb4 ERROR:  invalid input syntax for type smallint: "" at character 100
2024-02-24 20:35:13.973 CET [5632] jdb4adm@joomladb4 STATEMENT:  SELECT "update_id"
	FROM "j3ux0_updates"
	WHERE element = 'joomla' AND type = 'file' AND client_id = '' AND folder = ''

Call stack:

1 | () | JROOT/libraries/vendor/joomla/database/src/Pdo/PdoDriver.php:452
2 | Joomla\Database\Pdo\PdoDriver->execute() | JROOT/libraries/vendor/joomla/database/src/DatabaseDriver.php:1336
3 | Joomla\Database\DatabaseDriver->loadResult() | JROOT/libraries/src/Table/Update.php:109
4 | Joomla\CMS\Table\Update->find() | JROOT/libraries/src/Updater/Updater.php:309
5 | Joomla\CMS\Updater\Updater->getUpdateObjectsForSite() | JROOT/libraries/src/Updater/Updater.php:165
6 | Joomla\CMS\Updater\Updater->findUpdates() | JROOT/administrator/components/com_joomlaupdate/src/Model/UpdateModel.php:189
7 | Joomla\Component\Joomlaupdate\Administrator\Model\UpdateModel->refreshUpdates() | JROOT/administrator/components/com_joomlaupdate/src/Controller/DisplayController.php:80
8 | Joomla\Component\Joomlaupdate\Administrator\Controller\DisplayController->display() | JROOT/libraries/src/MVC/Controller/BaseController.php:730
9 | Joomla\CMS\MVC\Controller\BaseController->execute() | JROOT/libraries/src/Dispatcher/ComponentDispatcher.php:143
10 | Joomla\CMS\Dispatcher\ComponentDispatcher->dispatch() | JROOT/libraries/src/Component/ComponentHelper.php:361
11 | Joomla\CMS\Component\ComponentHelper::renderComponent() | JROOT/libraries/src/Application/AdministratorApplication.php:150
12 | Joomla\CMS\Application\AdministratorApplication->dispatch() | JROOT/libraries/src/Application/AdministratorApplication.php:195
13 | Joomla\CMS\Application\AdministratorApplication->doExecute() | JROOT/libraries/src/Application/CMSApplication.php:306
14 | Joomla\CMS\Application\CMSApplication->execute() | JROOT/administrator/includes/app.php:58
15 | require_once() | JROOT/administrator/index.php:32

richard67 - comment - 24 Feb 2024

Tomorrow I will try to find out what happens with scenario 5 on PostgreSQL (see my previous comment).

richard67 - change - 25 Feb 2024

The description was changed

richard67 - edited - 25 Feb 2024

richard67 - change - 25 Feb 2024

The description was changed

richard67 - edited - 25 Feb 2024

richard67 - comment - 25 Feb 2024

@SniperSister I've just tested all scenarios with success. At the end of scenario 7 I get an error which is not related to this PR (we downgrade on a current 5.1-dev where some modules have been converted to use service provider when we update to the older alpha package). So that's expected.

But there is something else which possibly could be improved.

Currently, without this PR, when your server doesn't meet both the PHP and the database requirements, information on both is shown.

With your PR in scenario 6, only the first fail is shown, i.e. the failed database requirement is only shown when the PHP requirement is fulfilled. I've tested with modifying the database driver to return a not sufficient version, in addition to the described hack for the PHP version fail.

Would it make sense to change that? Like now, people would first contact their hoster for the PHP version, and when that is fixed they will see the database thing and have to contact their hoster again.

richard67 - comment - 25 Feb 2024

After the last branch update to latest 5.1-dev, composer seems to fail in appveyor and drone unit tests. Maybe there were some changes in the 5.1-dev branch which somehow affect the composer changes of this PR here?

richard67 - comment - 25 Feb 2024

For my above comment #42799 (comment) about showing all failed constraints and not only the first one, I've made a PR: SniperSister#2

richard67 - test_item - 25 Feb 2024 - Tested successfully

richard67 - comment - 25 Feb 2024

I have tested this item ✅ successfully on 5585d32

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

richard67 - comment - 25 Feb 2024

I've successfully tested all 7 scenarios. In addition I've tested that updating from 4.4.3 to the package with this PR works with MySQL and PostgreSQL.

In scenario 3 I made additional tests for custom update URLs in the old XML format created by drone for pull requests:

PR for 4.4-dev: No update found. That's expected.
PR for 5.1-dev: Update to the patched package of that PR found as expected.

In scenario 6 I additionally tested a not fulfilled database requirement by hacking the getVersion method of the DB driver. I've noticed that in opposite to now without this PR, only the details of the first not fulfilled requirement are shown, but not both in case if both the PHP and the database requirement are not fulfilled. I've made a PR to the branch of this PR for an improvement about that, but it can also be done later with a follow-up PR in the CMS repo.

Important note for other testers: When testing scenario 7 on a current 5.1-dev branch or a 5.1.0-alpha4 with this PR applied, you will get errors after the update which come from the update not being up to date with latest changes in the 5.1-dev branch, so some admin modules which were converted to the new service provider structure will be downgraded to the before status and so cause failures. These are not related to this PR but to the update used for testing being out of date.
To avoid that you could use a 5.1.0-apha3 for testing.

richard67 - change - 26 Feb 2024

The description was changed

richard67 - edited - 26 Feb 2024

richard67 - test_item - 26 Feb 2024 - Tested successfully

richard67 - comment - 26 Feb 2024

I have tested this item ✅ successfully on 5585d32

I've tested scenario 6 again and can confirm that after the latest changes information on all failed constraints (DB and PHP versions) are shown.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

alikon - comment - 26 Feb 2024

is the CLI installation TUF aware ?

richard67 - comment - 26 Feb 2024

is the CLI installation TUF aware ?

It should be as it uses the model. But as I just could see the CLI doesn't show any details of failed constraints in scenario 6. But that was also the case without TUF, I think.

richard@vmubu01:~/lamp/public_html/joomla-cms-5.1-dev$ cli/joomla.php core:update:check

Joomla! Update Status
=====================

You are on the default update channel.
Your current Joomla version is 5.1.0-beta1-dev.

 ! [NOTE] New Joomla Version 5.1.100 is available.                                                                      
                                                                                                                        
 [WARNING] We cannot find an update URL                                                                                 
                                                              
richard@vmubu01:~/lamp/public_html/joomla-cms-5.1-dev$

richard67 - comment - 26 Feb 2024

Hmm, it seems not to be ready for the CLI. Trying the update (scenario 7) with cli/joomla.php core:update fails:

richard@vmubu01:~/lamp/public_html/joomla-cms-5.1-dev$ cli/joomla.php core:update

Updating Joomla
===============

Starting up ...
Running checks ...
Check Database Table Structure...
 1/9 [===>------------------------]  11%
                                                                                                                        
 [INFO] 7 database changes were checked.                                                                                
                                                                                                                        

                                                                                                                        
 [INFO] 46 database changes did not alter table structure and were skipped.                                             
                                                                                                                        

Starting Joomla! update ...
Processing update package ...
Downloading update package ...
Extracting update package ...
Copying files ...
 6/9 [==================>---------]  66%
In Folder.php line 65:
                           
  Source folder not found  
                           

core:update [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [--live-site [LIVE-SITE]] [--] <command>

richard@vmubu01:~/lamp/public_html/joomla-cms-5.1-dev$

richard67 - comment - 26 Feb 2024

In the update log after CLI update attempt:

#Fields: datetime	priority clientip	category	message
2024-02-26T12:11:22+00:00	INFO -	update	Test logging
2024-02-26T12:11:22+00:00	INFO -	update	Update started by user CLI (0). Old version is 5.1.0-beta1-dev.
2024-02-26T12:11:23+00:00	WARNING -	jerror	Error connecting to the server: 404

SniperSister - comment - 26 Feb 2024

That failure is not related to the actual TUF implementation but to the fact that the test release has invalid package URLs on 2 of 3 mirrors. The backend updater will fall back to the working one, the CLI updater fails on the first package. The information retrieval (that's the key part) works as expected.

bembelimen - change - 26 Feb 2024

Status

Pending

⇒

Ready to Commit

bembelimen - comment - 26 Feb 2024

Ready, Set, Go...

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42799.}

LadySolveig - change - 26 Feb 2024

Status	Ready to Commit	⇒	Fixed in Code Base
Closed_Date	0000-00-00 00:00:00	⇒	2024-02-26 16:30:51
Closed_By		⇒	LadySolveig
Labels	Added: RTC

LadySolveig - close - 26 Feb 2024

LadySolveig - merge - 26 Feb 2024

LadySolveig - comment - 26 Feb 2024

Thanks to all involved in this great project! ? ? ?

alikon - comment - 26 Feb 2024

so it's a new feature and/or it is a release blocker ? does it work from cli ?
what i'm missing ?
ah ... it is merged

Add a Comment

Older
Newer

Joomla! Issue Tracker - CMS

[#42799] - [5.1] TUF-based core updates

Summary of Changes

Testing Instructions

Preparation steps

Scenario 1: successful retrieval of a legit core update via TUF

Scenario 2: blocked retrieval of a malicious core update via TUF

Scenario 3: successful retrieval of a core update via a custom XML server

Scenario 4: successful retrieval of an extension update via the existing XML mechanisms

Scenario 5: reinstall feature is available

Scenario 6: Constraint information is availabel

Scenario 7: successful installation of a core update

Test Queries

Valid production metadata - MySQL and MariaDB

Valid production metadata - PostgreSQL

Invalid test metadata - MySQL and MariaDB

Invalid test metadata - PostgreSQL

Link to documentations

Kudos

Scenario 1

Expected

Actual

Scenario 2

Expected

Actual

Scenario 3

Expected

Actual

Scenario 1

Before PR

With PR

Comment

Scenario 2

Add a Comment