Feature PR-4.4-dev Pending

User tests: Successful: Unsuccessful:

avatar Septdir
Septdir
30 Nov 2023

Pull Request for Issue #42422

Summary of Changes

Testing Instructions

  1. Install old Joomla 4 version
  2. Apply patch
  3. Update Joomla

Actual result BEFORE applying this Pull Request

If Amazon was not available to you, the update was not downloaded.

Expected result AFTER applying this Pull Request

Update download and install

Votes

# of Users Experiencing Issue
1/1
Average Importance Score
5.00

avatar Septdir Septdir - open - 30 Nov 2023
avatar Septdir Septdir - change - 30 Nov 2023
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 30 Nov 2023
Category Administration com_joomlaupdate Libraries
avatar Septdir Septdir - change - 30 Nov 2023
Labels Added: PR-4.4-dev
avatar Septdir
Septdir - comment - 30 Nov 2023

@zero-24 Review code please

avatar pavluk pavluk - test_item - 30 Nov 2023 - Tested successfully
avatar pavluk
pavluk - comment - 30 Nov 2023

I have tested this item ✅ successfully on 08a6ed7

The update works without problems after applying the changes.
The update via Amazon still works for me.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42431.

avatar fiction13 fiction13 - test_item - 30 Nov 2023 - Tested successfully
avatar fiction13
fiction13 - comment - 30 Nov 2023

I have tested this item ✅ successfully on 08a6ed7


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42431.

avatar sergeytolkachyov sergeytolkachyov - test_item - 2 Dec 2023 - Tested successfully
avatar sergeytolkachyov
sergeytolkachyov - comment - 2 Dec 2023

I have tested this item ✅ successfully on 08a6ed7

Tested it successfully from 4.4.0 to 4.4.1


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/42431.

avatar sergeytolkachyov
sergeytolkachyov - comment - 5 Dec 2023

@rdeutz ping

avatar Septdir
Septdir - comment - 4 Jan 2024

@zero-24 @brianteeman I would like to solve the issue of update mirrors already.

So if something is wrong, write it down.

avatar Septdir
Septdir - comment - 9 Feb 2024

And?

avatar brianteeman
brianteeman - comment - 11 Apr 2024

This needs more detailed testing instructions. The better the instructions the easier/faster it is to test.

I have not tested it as I dont know how to deny access to amazon to trigger the error and therefore to validate the fix.

If you know of a way for me to do that (other than move to Moscow) please add those instructions to the pull request

avatar Septdir
Septdir - comment - 11 Apr 2024

@brianteeman Well, to check the operation of the update mirrors, you don't need to go anywhere. Although, if you want to take a tour of places where Amazon's storage is periodically blocked, it would make for a nearly round-the-world trip.

But to test the mirrors, it's enough to just connect another update server and start with the first broken URL.
Or traditionally accept the PR on the principle that it seems nothing is broken.

Another option is to look at the history of adding mirrors where there were detailed instructions.

avatar brianteeman
brianteeman - comment - 11 Apr 2024

Sorry but thats not a real test. Maintainers can disagree with me. Personally I only ever say I have tested something when I can replicate the original problem.

Or traditionally accept the PR on the principle that it seems nothing is broken.

No that is not the way it works as @Hackwar already explained to you in detail.

Find a way for me to replicate the issue and I will test it right away.

avatar Septdir
Septdir - comment - 11 Apr 2024

No that is not the way it works as @Hackwar already explained to you in detail.

PR #42422 One of the PRs that broke the mirrors was accepted within a day. And look at his Testing Instructions.

avatar laoneo
laoneo - comment - 16 Apr 2024

Thanks for your work @Septdir. But I see a lot of problems when we introduce all of the sudden a timeout. On slow connections where it worked before it has the potential to fail as it exceeds the timeout. That's why people came up with junk download solutions like #39280.

avatar HLeithner HLeithner - change - 24 Apr 2024
Title
[4] Fix Joomla update download mirrors
[4.4] Fix Joomla update download mirrors
avatar HLeithner HLeithner - edited - 24 Apr 2024
avatar brianteeman
brianteeman - comment - 24 Apr 2024

That's why people came up with junk download solutions like #39280.

chunk not junk ;)

avatar MacJoom
MacJoom - comment - 24 Apr 2024

Thank you for your contribution. We have discussed this matter in the maintainer group - we would like to have a solution with chunk updates and i will work on a solution based on #39280 - and we will change stuff in the updateservers (dealing with amazon problems). Closing this for now.

avatar MacJoom MacJoom - change - 24 Apr 2024
Status Pending Closed
Closed_Date 0000-00-00 00:00:00 2024-04-24 17:32:07
Closed_By MacJoom
Labels Added: Feature
avatar MacJoom MacJoom - close - 24 Apr 2024
avatar Septdir
Septdir - comment - 24 Apr 2024

Well, that's yours decision. If people can't properly update Joomla for 4 years, they might as well wait another 10.

However, regarding the implementation, I'll respond.

@laoneo the timeout here is not for downloading the file, but for the server response. No matter how slow the internet is, if the server doesn't send the headers within 5 seconds, its functionality is questionable.
In the official Joomla update manifest, there are 3 mirrors. The mirrors have long been non-functional (someone simply did an unset), but they are still listed in the manifest. Considering redirects, we get 10-15 seconds on the main mirror and about 3-6 seconds on the backup ones.
Only after that does the download begin.

The solution with chunk downloading (a very bad idea, in my experience) does not eliminate the need for mirrors. Simply put, there will be zero benefit from chunking if even the first part can't be received.

And this isn't about Amazon; there always needs to be a backup option.

Thank you for your attention.

avatar brianteeman
brianteeman - comment - 24 Apr 2024

Which mirrors should be removed?

avatar Septdir
Septdir - comment - 24 Apr 2024

No need to remove mirrors. They are supposed to work. That's the point.
If there are any issues with Amazon, we download from GitHub.
If there are issues with GitHub, we download from update.joomla.org.

The fact that Amazon doesn't always work for some people doesn't mean it always works for others.

Even if we replace Amazon with something else, it still doesn't give a 100% guarantee of the availability of update files. To approach 100% availability, three mirrors were created.

Since Joomla 3, three mirrors have been included in the manifests.
https://update.joomla.org/core/j5/default.xml
изображение

Yet, this line simply excludes them. (now used only first download link)

unset($this->downloadSources);

Paradoxically, the Joomla update manifest provides three download links, but Joomla itself deletes them.

In this PR, I simply reinstated the functionality of the mirrors and made some improvements to the old code.
Additionally, I set a timeout due to Amazon, which, instead of returning an error, just hangs for 30 seconds until the script crashes.

The timeout for receiving headers could be changed, although there's no point.

If you plan to go with the chunk method, then the mirrors will have to be integrated with this solution. For example, if the server delivering the file goes down after the first part has been downloaded, the second part should be retrieved from a mirror, and so on.

avatar Septdir
Septdir - comment - 24 Apr 2024

And yes, if we're talking about mirrors, language packs could also benefit from them.
Also, looking to the future, it would be good to collect feedback, for example, in the event of a core update error, a message could be displayed.

Inform us about the error, and you can manually install the update by downloading the package from:

  • downloads.joomla.org
  • GitHub
  • update.joomla.org
avatar Septdir
Septdir - comment - 24 Apr 2024

In summary, that’s about it.

  • I’ve told you the history and reasons behind the creation of the update mirrors in Joomla.
  • I’ve explained why they stopped working.
  • I’ve outlined the thought process behind this particular PR.
  • I’ve shared ideas for improvements and preventing future problems.

Have a nice day.

Add a Comment

Login with GitHub to post a comment