User tests: Successful: Unsuccessful:
Pull Request for Issue #36741 .
You can now tell Joomla Update to download the upgrade packages one megabyte at a time. This allows the update to download on very slow servers (see the linked issue).
Tagging @zero-24 @laoneo @HLeithner
First of all, the majority of Joomla sites are still on Joomla 3. It will take a long while for everyone to migrate to Joomla 4 and beyond. Let's not forget that there are still 3PD extensions which are still in the process of migrating to J4. Most sites can't migrate until it's clear which extensions can finish their migration and which are left behind to die.
Right now there are servers which fail to download the Joomla update package. If there is a security issue with Joomla 3 it will be very hard (if not impossible, depending on the site owner's skill level) to update these sites, exposing them to a security risks. We don't even provide an alternative method, like we do in Joomla 4 (you can update J4 over CLI where waiting even for half an hour for the update package to download is not a practical problem!). Therefore it makes sense to start by addressing this issue on Joomla 3 even though it's currently on security maintenance only.
If this approach works for y'all (and the production leadership ultimately decides that they cannot use CloudFlare as the default mirror which would solve this problem on most servers!) I can port this to Joomla 4.
Common instructions
manifest_cache
in the #__extensions
table for extension_id
700 to have a fake version, e.g. 3.10.10.libraries/src/Version.php
with PATCH_VERSION = 10Test 1
Makes sure that updates without any further configuration still work the same way as they did before.
Test 2
Makes sure that chunked downloads work.
Test 3
Makes sure the fine-tuning setting for the minimum time works.
You can redo this test with the minimum time set to 0 and observe that the download is much faster — unless your server throws an error because of too many requests…
Add the following to the help page for Joomla Update options.
The Fine-tuning tab's options control how Joomla Update will download the update packages before installing them on your site.
By default, Joomla Update tries to download the entire update package in a single page load. If your server has a slow network connection to the Joomla Project's update download servers it may time out before the download is complete, making it impossible to update your site.
If this happens to you — and only if it happens! — you should change the “Chunked Downloads” option to Yes. Joomla Update will now download 1 megabyte of the update package at a time, minimising the risk of your server timing out.
The “Minimum time between download steps (seconds)” controls how fast these 1 megabyte downloads will come. If the last attempt to download 1 megabyte of the update takes less time than this setting is configured to, Joomla Update will wait for the remainder of the time. Setting this to 0 will make downloads very fast but there is a chance that your hosting company's server protection may block you and break the updates. If you receive an error from your server while the download is in progress increase this setting. If you do not see any errors try decreasing this setting to make downloads faster. The default value of 3 seconds strikes a good balance between download speed and working around server protections which would block the update package download on most commercial hosting companies.
This PR introduces 10 new language strings.
This is for @HLeithner mainly and to explain why this option is disabled by default.
Every Joomla update provides a set of mirrors (alternate download URLs). The first mirror is the Joomla downloads site which runs a very old version of Akeeba Release System and is linked to an Amazon S3 bucket.
The items in this bucket are stored with Private ACL. ARS on the downloads site returns a pre-signed URL which allows only one specific HTTP verb: GET. There is no way to allow alternate verbs, such as HEAD.
To make chunked downloads possible we need to do a HEAD request on the mirror's download URL and follow all redirections. When we find the final destination we perform a final HEAD request and expect to see the HTTP Content-Length header which tells us how big is the file to download. Since the first mirror redirects to S3 using a pre-signed URL and the pre-signed URL does not allow HEAD requests we can not use the default mirror for chunked downloads. Therefore we will fall back to the next mirror which is GitHub and, if I recall correctly, GitHub wasn't very happy with Joomla using it for all its downloads.
In an attempt to alleviate the pain, I have implemented a simple workaround. If the user has NOT opted into chunked uploads I do NOT perform any HEAD requests. Instead, I simply get the list of mirror URLs and try them one after another, until one successfully downloads the entire file. Therefore the majority of users who are still on the single part download will be able to use the default mirror (hosted on S3) and all is good in the world.
If you want to make chunked downloads the default or only option you MUST do one of the following:
s3://
pseudo-URLs in the database to http://
or https://
URLs pointing directly to the file on the S3 bucket and change the item type to link
. While certainly possible, I do not see why bother when you have a cheaper, faster, more reliable method.Status | New | ⇒ | Pending |
Category | ⇒ | Administration com_joomlaupdate Language & Strings JavaScript |
Labels |
Added:
Language Change
?
|
Hopefully the new commit will do the trick.
Joomla 3 is using such old versions of the code quality tools that I no longer have a PHP version old enough to run them on when developing. So, I have to wait for Drone to finish the tests which takes quite a while.
Test 2 - after 99% completion, I got an error in the popup window. I logged out of the site, cleared the cache, logged in again, repeated the test - the same error, only now the execution breaks immediately, and not 99%.
The settings were set to 3 seconds.
5.5.5-10.5.15-MariaDB
PHP 7.4.29
Joomla "changed" to 3.10.10
@Kostelano Oops, I had made a change to develop this feature without the update actually running. I accidentally committed that in the repo I made the PR from. I fixed it now. Can you please retest?
The fix works. Test 2 was successful.
@Kostelano Thank you for retesting!
Now proceeded to 1 - I DISABLED the parameter, but when updating, firstly, exactly the same window is called as in test 2 (as far as I understand, the window should also be disabled ...?), secondly, exactly the same error, like the posts above.
Sorry, updated an old patch. Ehhhh
No worries :) To clarify, in all of the tests you will see the new download UI. However, with the parameter disabled it goes from 0% straight to 100%. With the parameter enabled it progresses through the bar. The current 3.10.11 update is around 12Mb and we download 1Mb at a time so you see the bar progress 100/12 = 8.33% at a time.
If you think this is unclear let me know, I can update the PR description.
I checked all 3 scenarios for the test. Everything works well. I am not sending a successful test yet, there will probably be some discussion and changes.
The only thing is that the download page of the package seemed a bit inconsistent in terms of design to me. I understand that this is a rather individual question. I would prefer the margins to fit the width of the screen (similar to Joomla's refresh page). Or, as an option, implement something similar as in Joomla 4 (in one line with execution, bytes, etc.).
@Kostelano The design of the page is better than the update page in 3 which is simply based on the fact that nobody wrote any CSS the last 10 years... What you see in the download page is the best you can do with the ten-year-old Bootstrap 2 still in use in Joomla 3. We can, of course, change the design of the update page to match. You can't do a single row without using Flexbox which was not an option in BS2 and I didn't want to add too much custom CSS.
Regarding the margins, I think that the 1200px max width is better than the insanity in Joomla 4. In J4 if you have a 48" ultrawide monitor and stupidly maximise the window it will happily take over an entire meter (1m) of screen width which is completely ludicrous. You can't read that information. It's silly. If we want Joomla to be silly let's add dancing unicorns and falling snowflakes on the update page like it's 1998 all over again, you know?
I have tested this item
Tested it successfully on a web hosting where the joomla update timed out before. They fixed the slow download partially so a call like wget https://downloads.joomla.org/cms/joomla4/4-2-2/Joomla_4-2-2-Stable-Update_Package.zip?format=zip
takes now only about 2 minutes. When running the update through the Joomla updater with this patch it finishes in around 20 seconds, but on j3, as I used the branch from this pr. So I welcome very much this change, even when I see here again no reason for an option. But in this case here it looks like it is needed, so no objection on this.
Hopefully our infrastructure can be changed that the chunked downloads will become the default behavior, as it looks much more professional and stable.
Good job!
I have tested this item
As I wrote above, all 3 scenarios for testing work well.
Status | Pending | ⇒ | Ready to Commit |
RTC
There is a line in the logs with information about fragment loading. The first fragment is not 1, but 0, then immediately 2. Something is wrong.
The range looks ok, there is no gap, so it seems only the fragment number 2 is counted wrong.
Labels |
Added:
?
|
Status | Ready to Commit | ⇒ | Pending |
Back to pending, see previous comments.
There is a line in the logs with information about fragment loading. The first fragment is not 1, but 0, then immediately 2. Something is wrong.
No, this is right. The first step is chunk -1 which tells the code to initialise the download. The next steps are 2, 3 etc for the byte range to be calculated correctly. As you can see the byte range is correct.
I cannot change the way chunks work because we'll end up breaking the download. In the grand scheme of things, having correct downloads is infinitely more important_ than having a number in the logs nobody will look at. So what I will do is change the message for the first chunk.
Labels |
Removed:
?
|
@Kostelano Could you test again just if the log is fine now? The rest hasn't changed, so if the log is ok I will set RTC again. But I don't have the time now to test it myself. Thanks in advance.
Sorry, it's a bit hectic around here with my daughter back from school. I mean to use max, not min.
Status | Pending | ⇒ | Ready to Commit |
RTC
I suggest different aproach, to keep thing easy:
Catch failed download, and show a message to User, suggest to download and install the package manually, the message should contain a link to release page.
How to catch failed download:
Before download are started the Updater set a session flag "updater doing download", and remove it after download are succeed.
After the server goes timeout and whole thing crashes, User will try refresh the page.
The updater check for the flag and if it still present then show a (warning) message to User: "Your ISP is Potato please try download and install packge manualy, and here is the link to download"
@Fedik Your approach is nonsensical and user-hostile! First of all, the user would wait forever until they are met with a server error page, thinking Joomla is broken. Then they need to go back to Joomla where they are met with a message that they need to download the file manually. However, their server is so crap that they can't even upload the (VERY) big Joomla update package. Therefore they are left with no update route. User's conclusion: Joomla sucks, we're moving to WordPress.
With my approach, even crappy servers can successfully download the update archive, however slowly, and run the update just fine. User's conclusion: My host sucks, but Joomla is a feckin' rockstar!
Also for your information: my approach didn't come from outer space. This is what I am doing in my own software (Akeeba Backup Professional) when you are downloading very large backup archives (think hundreds of MB to several GB big) from a remote storage back to your server. I don't tell people “aw, your file is too big, sucks to be you”. I download it a few MB at a time. It takes longer but it completes successfully.
The whole point is to provide the user with a VIABLE alternative, not tell them that it sucks to be them.
I agree with George, worth merging. We make sure that when we have a security update that people can download and install it.
As agreed on last and todays meeting there are no plans to merge this into J3 but only J4. There has not been an update posted here as we where awaiting a analyse of concerns raised by @HLeithner that I still have not got details on.
@zero-24 I have already addressed the concerns raised by @HLeithner in the “Important notes” part of my PR description when I submitted the PR over a month ago.
I also disagree that this shouldn't be merged on 3.10. This is where the pain is, not 4 (I have not even ported this code to 4 yet and, frankly, never will seeing that the project does not care).
Since this PR is only for Joomla 3 I am closing it with a mental note not to bother trying to fix Joomla.
Status | Ready to Commit | ⇒ | Closed |
Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2022-10-18 17:55:33 |
Closed_By | ⇒ | nikosdion | |
Labels |
Added:
?
|
@zero-24 I have already addressed the concerns raised by @HLeithner in the “Important notes” part of my PR description when I submitted the PR over a month ago.
I saw that this is why i wanted to wait for Harald to get back to me first with his concerns as that was two weeks ago he raised it to the team, I have not got any as mentiond above.
I also disagree that this shouldn't be merged on 3.10. This is where the pain is, not 4 (I have not even ported this code to 4 yet and, frankly, never will seeing that the project does not care).
The pain is more on J4 as there we have much larger packages that could be have issues with limited download capatatcies. Do not care is not fair but you know that too.
Thanks for your effort on this hopefully it can someday be ported to J4 and J5 so the update process can be improved for the future especially for regions/servers not having such a good uplink to all CDNs.
The pain is more on J4 as there we have much larger packages that could be have issues with limited download capatatcies.
@zero-24 As updating from J3 to the latest J4 version happens with the bigger update zip files and still many sites are on J3 for whatever reason I think it would be good to have that in 3.10.
The issue only occoured because of an wider and now solved network issue of an upsteam provider. As I dont expect many if any at all security releases of J3 it would might never reach real world sites. Aditionally after the merge it involves an massive upmerge issue as we have switched to psr12, and when even Nicholas cant easily convert the code to J4 i would not ask someone to do it in one major merge conflicted upmerge specificly on this critical part of the software.
@zero-24 You said:
so the update process can be improved for the future especially for regions/servers not having such a good uplink to all CDNs.
As I explained, the problem is that you are NOT using a CDN by default. You are using an S3 bucket which only serves files from a single region. This is in North America.
If you reorder the mirrors and put CloudFlare first then and only then will you be using a CDN. CloudFlare has edge nodes literally all over the world. Even if it's not in someone's country it's in an adjacent country (or state) and has more bandwidth than S3 which makes it by definition a much faster choice than using an S3 bucket. It's also free, which makes it all the more curious as to why the project is so adamant on using an S3 bucket to begin with. But that's neither here nor there.
The point is that this PR does not give a rat's ass about the mirror ordering or where your mirrors are served from. When chunked downloads are enabled it will go through every mirror until it finds one which responds to an HTTP HEAD request and use it. Therefore it's completely immune to any decision made about where updates are served from, something which is NOT true with the current version of Joomla Update which was written back when Joomla had exactly one download location (we're talking about 2012, Joomla 2.5.1).
So, what exactly is the hold up here? Why are we waiting for someone to tell us something that has nothing to do with the code in this PR? Seriously, I want to know. Help me understand.
The issue only occoured because of an wider and now solved network issue of an upsteam provider.
While this was the problem in this particular case, it's NOT the only case where downloads would time out.
PSTN / ISDN is still the primary or only way to get online even in developed countries e.g. rural parts of the USA. Satellite internet is not all Starlink; high-latency, low-bandwidth connections are the norm. Same goes for older mobile data connections, especially in Africa.
And yes, there are transient network issues which are not always under the control of an upstream provider. Between natural disasters (e.g. earthquakes severing fibre cables; been there), BGP SNAFUs, and international politics we can't assume that a fast Internet connection will be always available or that the download will take less than the PHP or server timeout, whichever comes first. We just can't. It is not a reasonable assumption.
Remember, this PR's Chunked Downloads feature is disabled by default and can be enabled at will when a user is affected by such a problem, regardless of its reason. Its purpose is to give a way out to people who can neither use the integrated update download, nor the Upload and Update features of Joomla Update. Ideally we'd add another option to have the update package stored somewhere on the server but that's another discussion and definitely NOT a user-friendly fallback, it's more of a Hail Mary approach, much like updating from the CLI in Joomla 4.
Aditionally after the merge it involves an massive upmerge issue as we have switched to psr12, and when even Nicholas cant easily convert the code to J4
Excuse me, what?!
The code in this PR did not come from nowhere. I am using this code in my commercial software since 2009, as I said in the original issue. Of course I can port it to whatever environment as long as it runs on PHP 5.4 or later! I am currently maintaining software which uses this code and which runs standalone, inside Joomla 3, inside Joomla 4, and inside WordPress. How exactly did you get the impression I cannot maintain the code I am actively maintaining for 13 years? What was the thought process here?
Redoing the PR for J4 is not hard at all; I just need to copy the methods I created and press CMD-ALT-L on phpStorm to change it to PSR-12. The interface code is far easier to port in Joomla 4 as it's already written in vanilla JS with a single jQuery function, $.ajax
, which can be trivially converted to Joomla.Request
. This was deliberate; I did not fancy doing it all over from scratch for Joomla 4. Instead of having to spend 10 hours I need to spend about 3.
The problem is, why should I? You are not rejecting this PR because of its code, you are rejecting it because you do not understand the issue it solves and believe it's only possibly caused by a transient upstream issue. If I port this to Joomla 4 you'll reject it on the same nonsensical reason and I will have wasted even more time.
Give me a good reason to port it to Joomla 4 and I'll do it this week. Give me none and I can't be bothered to waste even more of my time. It's as simple as that.
As I explained, the problem is that you are NOT using a CDN by default. You are using an S3 bucket which only serves files from a single region. This is in North America.
If you reorder the mirrors and put CloudFlare first then and only then will you be using a CDN. CloudFlare has edge nodes literally all over the world. Even if it's not in someone's country it's in an adjacent country (or state) and has more bandwidth than S3 which makes it by definition a much faster choice than using an S3 bucket. It's also free, which makes it all the more curious as to why the project is so adamant on using an S3 bucket to begin with. But that's neither here nor there.
Back then it was tested via curl on the update.joomla.org which is a CDN and even a cloudflare CDN and it was also very slow. So in that case it was not only an issue about S3 bucket alone. Not sure where that was posted IIRC it was Allon who tested it on a affected server.
My understanding is that in the mid to longterm we (The Project) want to switch to CF by default but I do not have a timeline for that.
The point is that this PR does not give a rat's ass about the mirror ordering or where your mirrors are served from. When chunked downloads are enabled it will go through every mirror until it finds one which responds to an HTTP HEAD request and use it.
Thats what happening right now too. So S3 would still be the first chooice as that HTTP Head request got a reply, S3 was not down or fully not reachable. But the download took to long and PHP timed out.
So, what exactly is the hold up here? Why are we waiting for someone to tell us something that has nothing to do with the code in this PR? Seriously, I want to know. Help me understand.
I do have the same question, raised to the people who told me there are issues I would do get feedback to. I have not got any feedback so I can not share the reasons here.
PSTN / ISDN is still the primary or only way to get online even in developed countries e.g. rural parts of the USA. Satellite internet is not all Starlink; high-latency, low-bandwidth connections are the norm. Same goes for older mobile data connections, especially in Africa.
I do agree there are reasons this can happen and this setup happens too for sites running on my laptop. But thats not usually something that a Server is using right? The client connection is not the limiting factor here. But the "Server" got directly to the CDN. Yes still there are servers that are not connected good to the internet and larger packages makes the issue worse.
The code in this PR did not come from nowhere. I am using this code in my commercial software since 2009, as I said in the original issue. Of course I can port it to whatever environment as long as it runs on PHP 5.4 or later! I am currently maintaining software which uses this code and which runs standalone, inside Joomla 3, inside Joomla 4, and inside WordPress. How exactly did you get the impression I cannot maintain the code I am actively maintaining for 13 years? What was the thought process here?
Redoing the PR for J4 is not hard at all; I just need to copy the methods I created and press CMD-ALT-L on phpStorm to change it to PSR-12. The interface code is far easier to port in Joomla 4 as it's already written in vanilla JS with a single jQuery function, $.ajax, which can be trivially converted to Joomla.Request. This was deliberate; I did not fancy doing it all over from scratch for Joomla 4. Instead of having to spend 10 hours I need to spend about 3.
You wrote:
I have not even ported this code to 4 yet and, frankly, never will seeing that the project does not care.
After reading this again its not that hard of an redjection to build it to J4 as it was in my head when i wrote my reply to Richard, so I do have to take that back. Looks like i missunderstand you here.
The problem is, why should I? You are not rejecting this PR because of its code, you are rejecting it because you do not understand the issue it solves [...]
This is wrong. I do reject the PR against J3 like I would redject most if not all PRs against a branch thats in security only mode and will mostlikly not reciver any update. It has nearly nothing to do with the actuall code in this PR here.
and believe it's only possibly caused by a transient upstream issue.
How long do we offer Updates via downlaods.joomla.org. When I remember correctly we only got two issue reports both very limited to an upstream issue and IIRC both geograhicly limited to austria.
If I port this to Joomla 4 you'll reject it on the same nonsensical reason and I will have wasted even more time.
This is why i asked within production how to proceed with this PR first and beeing told I would get feedback on what the concerns are. I have not got any feedback so asked today again, with no updates so posted what I have posted above redjecting it for J3 but not J4 given that my understanding was that we want such a thing implemented, but there are "concerns" that people want to look into first, atleast that was my understanding and I have been told that.
Give me a good reason to port it to Joomla 4 and I'll do it this week. Give me none and I can't be bothered to waste even more of my time. It's as simple as that.
I personally dont take decisions like this on J4 as you know. Such decisions will ultimatly be taken by the Release Leads of J4 like i did for J3.10. I personally would like a solution like that for J4 but would be interesting to know the concerns raised so they can be cleared up, when there are none any more even better. I personally would also like a solution that can be enabled by default and not "hidden" behind a setting. But TBH I have no deeper understanding of that thing and this level of network issues to say whether we should lock it behind a setting or not thats why I asked internally for feedback on this topic.
I do not have as much understanding as you all have here in this discussion, but it looks to me this would benefit Joomla 4 and the project forward. Thank you @nikosdion for another great contribution. What do you think @HLeithner ? Anybody with concerns using this approach going forward? Thank you @zero-24 for bringing this PR to my attention.
Back then it was tested via curl on the update.joomla.org which is a CDN
WRONG! When I submitted my PR the first mirror was redirecting to the S3 bucket. The next mirror was GitHub and only the last mirror was update.joomla.org.
So in that case it was not only an issue about S3 bucket alone.
I never said it was. I only said that in principle using S3 as the default mirror makes no sense because you are paying for it and it's generally slow. These are objective facts, not opinions.
My understanding is that in the mid to longterm we (The Project) want to switch to CF by default but I do not have a timeline for that.
This is irrelevant to this PR. As I told you, slow downloads can happen for a number of reasons. Moreover, as I told you, this PR doesn't give a fsck which mirror you are using and where it's located in.
Thats what happening right now too. So S3 would still be the first chooice as that HTTP Head request got a reply, S3 was not down or fully not reachable. But the download took to long and PHP timed out.
WRONG! Right now, without this PR, you do not have a timeout. If the HEAD fails the page dies with a PHP or web server timeout.
In my PR there's a hard timeout on the HEAD request for 10 seconds. If it doesn't complete within 10 seconds we go to the next mirror.
So, no, it does NOT work the same way. I understand the problem and have already worked around it.
I do have the same question, raised to the people who told me there are issues I would do get feedback to. I have not got any feedback so I can not share the reasons here.
So, we are not merging a PR most of the leadership says it needs to be merged because an unspecified person may or may not have said that they may or may not provide feedback on something which is unrelated to this PR? And then you ask me why I don't spend time porting it to Joomla 4? Well, that's why. It feels like I'm trapped in bureaucracy, like Asterix and the 12 feats (see "The Place that Sends You Mad" part).
I do agree there are reasons this can happen and this setup happens too for sites running on my laptop. But thats not usually something that a Server is using right? The client connection is not the limiting factor here. But the "Server" got directly to the CDN. Yes still there are servers that are not connected good to the internet and larger packages makes the issue worse.
You assume that Joomla only runs on a server on a datacenter rack. This is not the case. People don't upgrade sites live, especially when it's a minor —let alone a major!— version change. They take a backup, restore it locally, and work on the local copy. So, yeah, they are subject to these problems.
After reading this again its not that hard of an redjection to build it to J4 as it was in my head when i wrote my reply to Richard, so I do have to take that back. Looks like i missunderstand you here.
Apology accepted. No hard feelings.
This is wrong. I do reject the PR against J3 like I would redject most if not all PRs against a branch thats in security only mode and will mostlikly not reciver any update. It has nearly nothing to do with the actuall code in this PR here.
But this is a security adjacent PR! Right now there are sites which cannot apply security updates because the only update options we give them don't work. As far as I am concerned, fixing the updater is a security update in and of itself.
How long do we offer Updates via downlaods.joomla.org. When I remember correctly we only got two issue reports both very limited to an upstream issue and IIRC both geograhicly limited to austria.
You assume that people report issues correctly. If you look at the forum and the Facebook group people give stupid, irrelevant replies to update issues. That was the biggest motivation for me to redo the update extraction code in 4.0 and have good messages and good troubleshooting documentation. I was tired of people abandoning Joomla because they got horrid, irrelevant advice.
Also, for the last time, it's not about whether you use S3 or not. It's about does the software offer a fallback if the WHATEVER primary mirror has WHICHEVER problem? The answer right no is "he!l no, this sh!t just fails". I consider this neither a good security practice nor a good user experience.
This is why i asked within production how to proceed with this PR first and beeing told I would get feedback on what the concerns are. I have not got any feedback so asked today again, with no updates so posted what I have posted above redjecting it for J3 but not J4 given that my understanding was that we want such a thing implemented, but there are "concerns" that people want to look into first, atleast that was my understanding and I have been told that.
Therefore I should not even bother porting it to Joomla 4. Got it.
WRONG! When I submitted my PR the first mirror was redirecting to the S3 bucket. The next mirror was GitHub and only the last mirror was update.joomla.org.
You are right the first chooice is S3. What i mean is it was manually tested via curl. Manually via a curl call on the affected server directly to the update.joomla.org fallback URL.
So, we are not merging a PR most of the leadership says it needs to be merged because an unspecified person may or may not have said that they may or may not provide feedback on something which is unrelated to this PR?
I can tell you it was an agreement within the production meeting to not merge this PR into J3, even yesterday when I raised it again noone said "You are an idiot we defenetly need that in J3". And there is no "unspecified person" but Harald who raised concerns. I do value his concerns and would like to see them to be adressed it looks like he just did not had a chance to double check on that. J4.3 is not expected to be released tomorrow so we do have a few days to check and when required adress them.
But this is a security adjacent PR! Right now there are sites which cannot apply security updates because the only update options we give them don't work. As far as I am concerned, fixing the updater is a security update in and of itself.
When you had issues updating in the past you also have issues updating to this version and do have to do it manually any way, so there is 0 benifit. For J3 people right now, and as we do not expect an J3 release soon the new code would never reache any users site when there is no security issue found within J3 that would trigger an J3 security release be fore it finally goes into EOS mode. I do agree with J4 its getting a larger issue and therefor I would agree to see a solution implemented into J4.
You assume that people report issues correctly. If you look at the forum and the Facebook group people give stupid, irrelevant replies to update issues.
I do look there and have nerver seen an issue like this reported, other than to the times where we was aware of that austria provider issue. I do oftern take a closer look into updater issues specificily for the reasons you pointed out.
Also, for the last time, it's not about whether you use S3 or not. It's about does the software offer a fallback if the WHATEVER primary mirror has WHICHEVER problem? The answer right no is "he!l no, this sh!t just fails". I consider this neither a good security practice nor a good user experience.
The fallback was implemented for the case of S3 is fully blocked I do agree its not a perfect solution to all kinds of issues this is something we can improve on for J4 and beyond, I do fully agree.
Therefore I should not even bother porting it to Joomla 4. Got it.
Again. You have now even got a reply from the J4.3 release Lead who offered support for this feature, you have support from other leadership people including me who did not said its that we will never accept it but just not for J3. What you do with that information is up to you.
I agree with George, worth merging. We make sure that when we have a security update that people can download and install it.
Aren't we in a chicken and egg situation here. If there is a security issue then the impacted people wont be able to get it because they dont have this fix. So there would need to be a release just with this fix in advance of any possible security release in the future.
@brianteeman You can update Joomla Update as an extension before updating Joomla itself.
ok - forgot that - so it seems a no brainer to me to update the component. that way if there is ever a need for it the code is ready
@brianteeman You can update Joomla Update as an extension before updating Joomla itself.
Which suffer by the same issue as its also primalry hosted on S3, yes its even a smaler package so should suffer less. Standalone releases are something we should use with care as all code needs to be checked to be B/C with the codebase of 3.whatever baseline 3.x version we draw. I do agree when there are major issues with the updater like a core update fucked up something and many people are affected by such an issue, but that should always be the last resort of things pulled out. Specificly we should also do a stable release with that new updater so we are not going to have an mixed bag of code who is tring to update to J4 that we have to maintain.
@zero-24 Yes, slow downloads will be slow downloads and may time out. What are the workarounds we give people?
Upload and install / Upload and update
For extensions this is just fine because they are small. The biggest package (Joomla Update) is not even 300K, it will not be a problem.
Joomla Update packages are, however, ginormous. Like 22.5MiB or more. These would either be more than what most servers allow you (and it's a tall ask for users to change this) or they might take so long to upload that they are not practical.
Hence the need for a chunked download method for Joomla Update.
If you ask me, I would personally prefer it if all updates —including extension updates— were chunked downloaded. On the topic of extension updates I would also prefer it if each update was installed atomically instead of trying to install everything at once with all the problems which ensue (from timeouts to the far out issue I have reported and others have experienced, as it turns out). But this is neither here nor there. I will make a different RFC for it.
Yes, slow downloads will be slow downloads and may time out. What are the workarounds we give people?
Upload and install / Upload and update
Manually upload the packages like they did in the past when they would actually have issus with the J3 packages already.
Joomla Update packages are, however, ginormous. Like 22.5MiB or more. These would either be more than what most servers allow you (and it's a tall ask for users to change this) or they might take so long to upload that they are not practical.
J3 is 11.9 MB. J4 is much bigger I do agree.
Hence the need for a chunked download method for Joomla Update.
Yup I do agree.
If you ask me, I would personally prefer it if all updates —including extension updates— were chunked downloaded. On the topic of extension updates I would also prefer it if each update was installed atomically instead of trying to install everything at once with all the problems which ensue (from timeouts to the far out issue I have reported and others have experienced, as it turns out). But this is neither here nor there. I will make a different RFC for it.
I do agree.
Can we reopen this PR? This will benefit Joomla BIG time. What do you think @nikosdion @HLeithner @wilsonge @zero-24 (sorry, not mentioning everyone but I want to bring attention back to this PR and everyone is welcome to comment)?
Status | Closed | ⇒ | New |
Closed_Date | 2022-10-18 17:55:33 | ⇒ | |
Closed_By | nikosdion | ⇒ | |
Labels |
Removed:
?
|
Status | New | ⇒ | Pending |
@obuisard Sure, I can click the button to reopen the PR. However, a few major points:
Thank you @nikosdion. I understand your points and they are valid indeed. If not for 3.10, the problematic remains the same in Joomla 4 (if I understand correctly). I am hoping to get feedback from maintainers, and I certainly have no intention of wasting your time on this issue.
@obuisard Yeah, I understand.
Honestly, if the leadership tells me yeah, this is a good approach, we want it in Joomla 4 I will drop everything and work on it. I just need to know I am spending time for a good reason, you know? :)
Totally understand @nikosdion.
Can we reopen this PR? This will benefit Joomla BIG time. What do you think @nikosdion @HLeithner @wilsonge @zero-24 (sorry, not mentioning everyone but I want to bring attention back to this PR and everyone is welcome to comment)?
I mean I've said before I think it's worth and I stand by that. But I'm not on production so no longer my call :)
Status | Pending | ⇒ | Closed |
Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2022-12-07 12:24:42 |
Closed_By | ⇒ | nikosdion |
I am closing this PR against J3 since it's clear there is no way it will be merged. If you ever change your mind, the code's here.
Thanks :)
@nikosdion Drone reports code style errors: https://ci.joomla.org/joomla/joomla-cms/57975/1/3