? ? Pending

User tests: Successful: Unsuccessful:

avatar zero-24
zero-24
3 Nov 2019

Summary of Changes

Allow to offer a different reinstall url than the download URL

Testing Instructions

First test

  • install an outdated version
  • apply this patch
  • make sure the update works
  • re-apply this patch
  • make sure the reinstall works too

Seccond test

Expected result

You can use a different reinstall url than the download url that allows us on updates to ship only the patch update and not always the full update. This should reduce the traffic we get on downloads.joomla.org as well as for the users that don't need to download the full package on any update, this should also make the updates faster for that reason.

Actual result

You can only use the download url and this forces everyone to download the full package on any update.

How is this archived?

<updates>
[...]
	<update>
[...]
		<downloads>
			<downloadurl type="patch" format="zip">https://downloads.joomla.org/cms/joomla3/3-9-12/Joomla_3-9-x_to_3-9-12-Stable-Patch_Package.zip</downloadurl>
			<downloadsource type="patch" format="zip">https://github.com/joomla/joomla-cms/releases/download/3.9.12/Joomla_3-9-x_to_3-9-12-Stable-Patch_Package.zip</downloadsource>
			<downloadsource type="patch" format="zip">https://update.joomla.org/releases/3.9.12/Joomla_3-9-x_to_3-9-12-Stable-Patch_Package.zip</downloadsource>
			<reinstallurl type="full" format="zip">https://downloads.joomla.org/cms/joomla3/3-9-12/Joomla_3.9.12-Stable-Update_Package.zip</reinstallurl>
			<reinstallsource type="full" format="zip">https://github.com/joomla/joomla-cms/releases/download/3.9.12/Joomla_3.9.12-Stable-Update_Package.zip</reinstallsource>
			<reinstallsource type="full" format="zip">https://update.joomla.org/releases/3.9.12/Joomla_3.9.12-Stable-Update_Package.zip</reinstallsource>
		</downloads>
[...]
		<sha256 type="patch">4bf360e449049efeab3a1f7ae24f49311a611bb4564e5fd30dce22ac4fced261</sha256>
		<sha384 type="patch">41f0596cdcf6bed2e5fda7f6d5092320fab75b9a8fbd3e1ae2f01d8c6f5a8ab73190341f475dd2aee21f63ba0316581e</sha384>
		<sha512 type="patch">4afd4870ff0433e4a4dd96cacb913758c4958d19431b45b3aa835b6bc5aa5286e840709502d4e9bb00801e727f9712b8b28cd8379ef18c629489ce2bd5ca57f7</sha512>
		<sha256 type="full">39c24efc5a4a811359f92d261d20426ee392cc1799af15dce6fe14fc7c49de20</sha256>
		<sha384 type="full">896d32d51b9573b74c4b2f01adc7c5c28db172ce5f2c05c97278a418dd14b45fbe164c789337550f2458716f4bae4bd1</sha384>
		<sha512 type="full">8b9216cd381f07b675bdc3fafdad6fc30725555e652f02bcdf8faa31225cca81b9fc9b7a24fd33805185b8065c9e06c51dd7583c6b9e7d050d823bafe902d04d</sha512>
[...]
	</update>
</updates>

Documentation Changes Required

The new reinstallurl and reinstallSource tags as well as the type attributes for that tags as well as the hash tags needs to be documented and will be by me. Label added.

As requested by @HLeithner and @wilsonge

avatar zero-24 zero-24 - open - 3 Nov 2019
avatar zero-24 zero-24 - change - 3 Nov 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 3 Nov 2019
Category Administration com_installer com_joomlaupdate Libraries
avatar zero-24 zero-24 - change - 3 Nov 2019
Labels Added: ? ?
avatar mbabker
mbabker - comment - 3 Nov 2019

This is reaching a point where I no longer agree with what you are doing with the definition of an update server, which is by design meant to provide a system for updating extensions. It was never suitable for core updates, and hacking in features to support core related features that extensions cannot use is just a bad idea.

This reinstall mechanism needs to be localized into the update component, not built into the updater library. It's just architecturally wrong.

avatar zero-24
zero-24 - comment - 3 Nov 2019

This reinstall mechanism needs to be localized into the update component, not built into the updater library.

Can you give a bit more detail what you want this to be? Maybe an dedicated xml/json file that the core fetches that does not go via the update xmls? Or what do you mean by needs to be localized in the updater component?

It's just architecturally wrong.

Well it was not me who build that feature this way.

avatar zero-24
zero-24 - comment - 3 Nov 2019

Do you think we could use the signature downloads API endpoint?

As we know the version we want to reinstall and this endpoints provides the hashes and download file names.

Or should we be carefull with building that endpoint directly into the cms?

Maye we can expose an dedicated endpoint for the reinstall infos?

avatar wilsonge
wilsonge - comment - 4 Nov 2019

I don't think the reinstall feature should point to the patch files? The update definitions can. But reinstall means reinstall the full thing - not just the last update - all core files should be overridden which effectively means using the full update file

avatar zero-24
zero-24 - comment - 4 Nov 2019

I don't think the reinstall feature should point to the patch files?

Yes this is the main problem. That we have two packages one, the full, for reinstall and the patch for the normal update.

So for now we have to always use the full package on update and reinstall.

avatar mbabker
mbabker - comment - 4 Nov 2019

I would rather see core updates decoupled from the Joomla\CMS\Updater library. It is a library for updating extensions. Sure, it does provide a standardized interface for defining a reusable manifest for supplying update information, but there are features present in the library that aren't fully implemented and there are features present in the library that are borderline unusable by extensions but because core uses that system the code lives there. This "reinstall" feature has no business being in the updater library, it is a workflow very specific to the core CMS and should exist in a spot that only applies to the core CMS, which would be com_joomlaupdate right now. Just look at the diff here, how would an extension be able to use this feature? If your answer is "they aren't", then you've written the code into the wrong spot; library level code is code that implements a feature that should either power the core of Joomla or be extended upon by the ecosystem.

Truthfully, I've always disliked the need for core to be able to parse a manifest on its own which means that adding new features to restrict availability update relies on core being updated (whereas with other platforms, basic telemetry is sent to an API server (which should need no more info than the statistics system, and that info need not be logged at all) and the server decides what update to offer you, but then you get into this whole debate that too much info is transmitted out to decide if updates are available).

I would like APIs built into joomla.org to expose information that the CMS and ecosystem in general could readily use, but that would mean that the project would actually have to dedicate resources to supporting APIs from joomla.org (I'm going to call this doubtful, there are no APIs available anywhere on .org, services are not built with the use case of a public API in mind (see the reverse engineering inside the Install from Web system and the lack of APIs for things like the events feed), the RSS feeds are hidden behind FeedBurner which is another broken and abandoned service but nobody puts the effort into replacing, people have at various times tried to subvert the use of AWS for storage because "we're Joomla, why are we paying for a service!?" which to me means that there is no long term stability for possible API lookups, and there are still no resources available for the shutdown of JoomlaCode).

That we have two packages one, the full, for reinstall and the patch for the normal update.

At some point the entire packaging and publishing workflow should also be revisited. I dare suggest it should almost be possible for someone to trigger a CI job that just builds packages and publishes them somewhere so that an optimized upgrade package can be made available regardless of release (this concept of "upgrading between minor versions requires the full upgrade package" is kind of garbage as it is no more difficult to create a patch package from 3.8.13 to 3.9.13 than it is to create a patch package from 3.9.0 to 3.9.13, the only difference is the number of files included). But that requires ditching the use of the extension update library which requires hardcoded URLs to a resolvable ZIP package to be used regardless of what version you are coming from (or maintaining an XML manifest that is way too complex) and using an API that can be aware of the scenario of "I am on version 3.9.2 and would like to upgrade to 3.9.13, please serve me the most efficient patch package possible" (and if we'd get rid of the copyright dates in every file, something which never made it to legal review regardless of the number of times I asked for it, you wouldn't have a patch file every year that contains the entire CMS). You just have to sort the logistics out of ensuring that the key packages are available for download on a public facing website (key packages being full install and full upgrade, no need to list all patch variations out).

avatar zero-24
zero-24 - comment - 4 Nov 2019

I'm with you and you know I don't like the update system how it is today.

Truthfully, I've always disliked the need for core to be able to parse a manifest on its own which means that adding new features to restrict availability update relies on core being updated

I would like APIs built into joomla.org to expose information that the CMS and ecosystem in general

Did you ever saw this file? (GitHub don't like yaml files so this needs to be a txt file)
swagger-joomla-update.txt

This is a very basic idea not implemented yet but maybe a step in the right direction, or is this total wrong again?

I'm happy work with you on that this here was just a idea to move this issue forward in the short term but I get the point of that making no sense at this point.

avatar zero-24 zero-24 - change - 18 Dec 2019
Status Pending Closed
Closed_Date 0000-00-00 00:00:00 2019-12-18 20:28:59
Closed_By zero-24
avatar zero-24 zero-24 - close - 18 Dec 2019

Add a Comment

Login with GitHub to post a comment