User tests: Successful: Unsuccessful:
Downloading the Joomla update package runs on some web sites into a timeout for the main Joomla download server and crashes the update. This should be catched and then the fallback download servers used instead.
The timeout is set to 30 seconds. Not sure if this is too short on slower hostings. Any recommendations?
[UPDATE]: As this here is more of a hot fix. End goal should be to fetch the update package in an ajax request as came out in the discussion of this pr.
Status | New | ⇒ | Pending |
Category | ⇒ | Administration com_joomlaupdate |
This should be catched and then the fallback download servers used instead.
Dont know what they are but probably should also display a message with advice how to manually download the update and then update from that file
The intention is to fallback to github and update.joomla.org first but yes on a follow up PR we could add a message when all three sources failed to provide the update.
sadly aws is not really fast to all isps, I have servers seen which downloads only with 153KB/s from aws US which means 2m 46s for download. (github mirror needs 8,9 seconds, update.joomla.org needs 5.1s)
Since the timeout parameter implements a hard timeout (complete operations have to be completed within 30 seconds) means the connection needs to have a minimum bandwidth of 9MBit. That doesn't sound much for most countries, but looking at the server mentioned it's located in Austria at Magenta (daughter company of T-Mobile Germany).
So the right value should be based on the minimum bandwidth we would allow to download the update.
Better solution would be to complete the change from AWS as primary download source to update.joomla.org (which is behind cloudflare) and drop (our own AWS) mirror. At least that should make it better for countries outside the US.
Edit: Changed the cloudflare part
So 30 seconds would work here as with this PR we would fallback to github and update.joomla.org which are well below the 30 seconds we set as limit or i'm missing something?
An change to cloudflare can and in my understanding is already beeing activly worked on but should not prevent us from patching this issue here right?
I have tested this item
In my opinion 30sec is very small, especially for joomla 4 where package much bigger in size.
Maybe something around 3-5min at least.
are you measuring the time to start the download or the time to complete the download
the time to complete the download
In my opinion 30sec is very small, especially for joomla 4 where package much bigger in size.
Maybe something around 3-5min at least.
Yes I get the point but the problem is the maximal timeout of the server right? The timeout we are faceing where the webserver of the site times out (because AWS took to long)
are you measuring the time to start the download or the time to complete the download
To establish the connection and to complete it:
I have tested this item
Tested with php8...
And this is what exactly happens. So the download takes ever and on some point the max execution time of the web server, where Joomla is running on, kills the process with a timeout error. It is an issue between your web browser and the Joomla server and not between the Joomla server and the download server from joomla.org. As if the web server would allow a max execution time of, lets say an hour, then the update will succeed.
Status | Pending | ⇒ | Ready to Commit |
Labels |
Added:
?
|
RTC
seems very wrong to me
seems very wrong to me
What seems very wrong to you? We have two different times to cosider the time it takes to download from AWS and the time our webserver allows us to run. Right now we dont limit the time to download so the webserver kills the process. With this patch we limit it ourself so we can fallback to other locations when AWS is not as fast as required.
The approach is to a bit too simple and doesn't consider default php max_execution_time which is per default 30 seconds, Webhosting company may change this to a higher value.
The problem is that you may help websites getting a timeout because they have only 60 seconds to download but kill webserver hat needs 70 seconds to download....
I would suggest a combination of 3 values (most of them only work with curl I think and may nott exposed to the joomla http transport shim).
set CURLOPT_CONNECTTIMEOUT to 5 seconds
set CURLOPT_LOW_SPEED_TIME to 5 seconds
set CURLOPT_LOW_SPEED_LIMIT to 10 Mbit/s
Setting the minium speed to 10 Mbit/s means that Joomla needs about 24 Seconds to download. if the connection rate is slower then this we would switch to the next fallback host which hopefully is faster then the first one and we still have enough time to download the package. See my example above, 2 minutues for AWS-US and 9 seconds for github and 5 for upload.joomla.org
Of source this would force a internet connection from minimum of 10 Mbit/s
Finally a voice of sense.
Setting the minium speed to 10 Mbit/s means that Joomla needs about 24 Seconds to download.
Have you tried to update joomla on a local development machine on a 256k connection?
What seems very wrong to you? We have two different times to cosider the time it takes to download from AWS and the time our webserver allows us to run. Right now we dont limit the time to download so the webserver kills the process. With this patch we limit it ourself so we can fallback to other locations when AWS is not as fast as required.
You are not considering the connection speed
What seems very wrong to you? We have two different times to cosider the time it takes to download from AWS and the time our webserver allows us to run. Right now we dont limit the time to download so the webserver kills the process. With this patch we limit it ourself so we can fallback to other locations when AWS is not as fast as required.
You are not considering the connection speed
What connection speed? Is the connection speed from me to my server actually relevant? The only thing thats send between me and my server is the html to render the Joomla Update site right? The update itself will be downloaded by the server without any data send to my local pc or i'm missing something?
... and if your server is local?
... and if your server is local?
Ok good that i asked i hadn't that in mind. Hmm so we actually have to check the max_excecution time too? Whe its set to 0 (infinit) its easy but i dont know how we could best do it for the other settings. Cause we cant just take whats set there as we need a bit more time after the download.
finally
That's why I asked you
are you measuring the time to start the download or the time to complete the download
This should be catched and then the fallback download servers used instead.
Dont know what they are but probably should also display a message with advice how to manually download the update and then update from that file
All you can do is to display the message. If the download takes more than your arbitrarily defined time to download from one server it will almost certainly time out on any other server.
You need to go back to first principles. What is the problem I am trying to solve. Do I know anyone with experience in this specific area that can help/advise/guide
All you can do is to display the message
I think this is another side of problem, the downloads start on "form submit", or I am wrong?
And if User will get Apache timeout, you cannot catch this error and display correct message. User will see white page or page with default server error.
The process could be like this:
So to move on here I propose to merge it as a quick fix and then open an issue which will address a new feature regarding the download in an ajax request.
Please dont. You are breaking more than you are fix
What is broken?
Already stated.
If the download takes more than your arbitrarily defined time to download from one server it will almost certainly time out on any other server.
As you can see for the actual problem this was not the case. The download to AWS took ages and the one to github fast fast.
This pr limits the time to fetch the package and when all (including the fallback) are failing then the checksum check here should return false . Which will lead to an exception as you can see here that the checksum is wrong. So actually the behavior is correct.
What can be discussed is if the timeout of 30 seconds is too restrict.
Please provide test instructions to validate that after 30 seconds instead of crashing the download is switched from aws.
Please provide test instructions
With assumption you use local server, I think it can be tested like next.
Temporary edit "hosts" file of your OS to redirect downloads.joomla.org to localhost:
127.0.0.1 downloads.joomla.org
Or block this domain in your router.
Then try update (probably need some dev channel, not sure here, or install older version first).
It should wait 30 sec, then switch to one of:
github.com/joomla/joomla-cms/releases/download
update.joomla.org/releases
And finish the update.
and without this pr you are saying that it will never switch to one of the other servers?
If so then I can not replicate the original problem so cannot test this "fix"
How have the two testers actually tested that this works.
Because the actual problem is not that downloads is not reachable. Downloads is reachable the underling US AWS is slow on some hosts as described above by Harald. So the test would be requiringing to have an affected host.
Or simulating an reachable but very slow replying AWS.
well I tried simulating it using network throttling and did not have any issues even when throttling down to a 50kbs connection
How and more important, where, did you set network throttling?
I tried it in two places.
Chrome browser
Thats the thing that does not have an affect here as its between the client and the server. The issue here is between your server (even when that is localhost) and aws where crhome does not have any impact.
browserstack
Dont know the details how browserstack works but from its name it does not sound like that you acutally limit the bandwith that the hosting server has aviable to download the package.
The other way arround could be a changed update server where you point to a never responding server I'm not sure how such thing would have to be setup.
But i understand the point that there could be issues for localhost sites not getting the package within 30 seconds but they have very long max_execution times (even 0=infinite) when the local PC (which would be the server in that case) not get any of the three packages within the 30 seconds the update fails.
The aviable max_execution time has also be devided by three given that we try three different locations too right? Could that be a workaround? What about checking that time and try to set that? And for example do not limit it when its set to 0? What happens when AWS does not come back at all (or runs infinite) at somepoint we should step up and direct the users to a different mirror right?
The other way arround could be a changed update server where you point to a never responding server I'm not sure how such thing would have to be setup.
See my previous comment
Can as well, use non existing IP like 196.100.100.100 for hosts
Ok @brianteeman, you still didn't understand the issue here. Follow the steps and then you will see the timeout:
It will timeout as the update is fetched from https://joomla.digital-peak.com/slow.php?file.zip where I do a sleep of 99999 seconds. When you apply the patch, then the update will work. For Joomla 4 you need the patch from #36742.
Clearly not just me as others have said to do X to test which was not correct
And could you reproduce the issue?
Sorry I was watching the football.
Trying to follow your instructions
trying again on a new server
same as before with point 4 but now update found in point 6
Try to update - eventually get a 500 error
applied patch from 36742
Try to update - eventually get a 500 error
Server info
max_execution_time 60
php 8.0.10
Sorry I was watching the football.
4. Check that in the table j_updates the entry with the id 42 has https://joomla.digital-peak.com/joomla.xml as details url There is no entry in this table that matches. There are only language update sites. Did you mean j_update_sites and the location url?
Didn't know that football is still a thing in England
Did you check it with PHPmyAdmin? If yes, then you probably need to got to the second page as it is the last entry in the table. And make sure you have installed version 4.0.5.
Sorry - I must have been distracted by the football. The record is present. Just retested everything again on a different net connection and still just get the 500
Where the patch is applied, can you echo there the url and die. What is first tried for a connection?
Ideally, Joomla would implement a staggered file download just like I implemented since 7 years ago in FOF 3: https://github.com/akeeba/fof/blob/fof-3.x/fof/Download/Download.php#L263
This can be used to download a file one MB at a time and reassemble it at the server without ever timing out, see https://github.com/akeeba/kickstart/blob/development/source/urlimport/feature.urlimport.php for the code I am using in Kicsktart Professional to import several Gigabyte big files since tern or eleven years ago.
All these problems are solved years ago.
However, here's an interesting thing to think about.
Staggered downloads only make sense when using the web interface. When using the CLI you should use the current method. Therefore you would need to different extension update paths which is impossible in the current architecture. And if you are going to go through this pain, why not implement the whitepaper I had written back in 2012 about making extension updates through the web interface resilient against slow hosts? That's something for @HLeithner to think about as any major change in that are would come with Joomla 5.
@HLeithner @bembelimen BTW, if you need me to make a PR for chunk downloads of the Joomla Update package for Joomla 4.1 I can do that. This wouldn't touch extensions updates but it would fix downloading the really big Joomla Update package. Between that and the other two PRs I've made (one merged, one pending) that'd be a pretty major upgrade to Joomla Update's reliability.
If you guys want me to do that please tell me today; I might be able to work on that this weekend.
This is some of the most important code in joomla. If any pr breaks the update process then we're screwed. Thats why I am being so insistent that it needs to be a verifiable fix
No problem. In the file where you applied the patch can you change the line to
$result = HttpFactory::getHttp([], ['curl', 'stream'])->get($url,[],30);echo $url;die;
.
So this should print the url to download after 30 seconds. And without the patch it will never return.
So this should print the url to download after 30 seconds.
It doesn't - already tried it but did it again just to be certain
And without the patch it will never return.
It is the patch ??
Do you have curl installed? Looks like the timeout doesn't work for some unknown reasons. I'm lost then.
@HLeithner @bembelimen BTW, if you need me to make a PR for chunk downloads of the Joomla Update package for Joomla 4.1 I can do that. This wouldn't touch extensions updates but it would fix downloading the really big Joomla Update package. Between that and the other two PRs I've made (one merged, one pending) that'd be a pretty major upgrade to Joomla Update's reliability.
If you guys want me to do that please tell me today; I might be able to work on that this weekend.
of course that would make much sense and the upgrade process much user friendly, but I think the target would be 4.2, for 4.1 it's too late, so no hurry.
@roland-d and @fancyFranci is responsible for 4.2 just for tagging this.
@brianteeman use a local proxy with throttling
curl_setopt_array($ch, $options);
) and add this line: $options[CURLOPT_PROXY] = '127.0.0.1'; $options[CURLOPT_PROXYPORT] = '8888'; $options[CURLOPT_SSL_VERIFYPEER] = 0;
now your local joomla installation should use the proxy to download files from the internet, for better overview what gets downloaded use the "Sequence"-tab in Charles proxy. You can also change the download speed in top menu "proxy->throttle settings"
@HLeithner Agreed. I will work on the PR over the next few days. I am now moving out of your way here.
Thanks @HLeithner
Using your instructions and throttling as I was able to confirm that without this PR it would result in a 500 error when the local server max_execution time was reached.
After applying the patch I can now confirm that after 30seconds the updater switched to the secondary source of github.
So I am happy to assume* that if I reconfigure charles to only throttle amazon it will indeed fallover to github and complete
I say assume because I didnt manage to get charles to only throttle aws
Nice you got it sorted somehow, then I delete my public files.
@bembelimen Can the „Updates requested“ label be removed now as things have been discussed above?
Sorry to say but I'm not convinced that this is the right way, because people in countries with slower uplink but longer execution time get lockout from the this update process.
Thats what I started off saying
also if the server has a max-execution time of 60 then with this code you can never fall back to the third source
Sorry to say but I'm not convinced that this is the right way, because people in countries with slower uplink but longer execution time get lockout from the this update process.
It is easy to say that, better would be to propose an alternative.
It is easy to say that, better would be to propose an alternative.
I did?
The approach is to a bit too simple and doesn't consider default php max_execution_time which is per default 30 seconds, Webhosting company may change this to a higher value.
The problem is that you may help websites getting a timeout because they have only 60 seconds to download but kill webserver hat needs 70 seconds to download....
I would suggest a combination of 3 values (most of them only work with curl I think and may nott exposed to the joomla http transport shim). set CURLOPT_CONNECTTIMEOUT to 5 seconds set CURLOPT_LOW_SPEED_TIME to 5 seconds set CURLOPT_LOW_SPEED_LIMIT to 10 Mbit/s
Setting the minium speed to 10 Mbit/s means that Joomla needs about 24 Seconds to download. if the connection rate is slower then this we would switch to the next fallback host which hopefully is faster then the first one and we still have enough time to download the package. See my example above, 2 minutues for AWS-US and 9 seconds for github and 5 for upload.joomla.org
Of source this would force a internet connection from minimum of 10 Mbit/s
Then make a pr and I close this one. I mean it fulfilled it's purpose, it saved Switzerland.
Hey Allon, I did propose a better solution and I’ll make the PR.
As I said, the problem you’re trying to solve is a solved problem and what you are doing is not the right way to deal with it. It’s not even a reasonable workaround. It’s a bad hack which will cause more problems than it solves.
Right now we have a minority of users who have a problem downloading updates. They can always download the update manually and use the upload and upgrade feature I added back in Joomla 3.4.
What you are doing will lock out many live sites and pretty much ALL locally hosted sites from upgrading with a single click EVEN THOUGH this worked for them just fine before.
Breaking something for at least 100x the number of people you’re fixing it for is NOT a solution. This is mass distributed software. People rely on our work. We can’t go around and break stuff just to serve our own narrow use case, consequences be damned.
The point you raised is valid and needs addressing. This PR is not the way to address it. I will make a PR once I feel better. Wait for a week or less. Don’t break things in the meantime.
I tried saying that politely before but since you seemed to ignore me I have to be more forceful. I don’t like doing this. It’s upsetting to me too.
Status | Ready to Commit | ⇒ | Closed |
Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2022-01-22 12:35:34 |
Closed_By | ⇒ | laoneo | |
Labels |
Added:
?
?
|
I do not have a problem to close as it was indeed a hack. But we needed fast a solution and this was one.
This is again an issue on some Swiss hosters. We need to find a solution for this!!
/me wonders why only switzerland
@laoneo @HLeithner I have talked to both of you about it. This is ultimately NOT something to fix with code, this is an infrastructure issue.
If you want me to implement chunked download for the updates I can do the PR. Do note that if the problem is as bad as Allon and Harald say — a few hundreds Kilobits per second — even that might not be enough as 1MiB (the lowest chunk we can go for without causing network issues…) would take several seconds. Also, I would only make this an optional feature. For people who can use the regular one–shot download it's actually better to stick with it!
Right now, Joomla is trying to use either a download from an Amazon S3 bucket or through GitHub releases (which also uses an Amazon S3 bucket).
The problem is that Amazon S3 buckets are localised, they only exist in the [region](https://docs.aws.amazon.com/general/latest/gr/s3.html_ they are created in. For example, an S3 bucket in the eu-north
region only exists in a few datacenters around Stockholm. If you try to reach that from Tokyo or, worse, from the Pacific the data transfer speed will suck incredibly and appear almost stalled.
At the very least, Joomla should use multi-region access for its bucket or use an Amazon CloudFront CDN. I can't tell you which one would work better for you without having the number of requests and transfer volume but you having all that information can use Amazon's cost calculators to figure it out.
I thought that we took advantage of an aws grant to cover the costs?
@nikosdion despite it is an infrastructure issue, somehow we should be able to handle this, even when it is a simple download speed information or whatever. I know with the current approach, where we download the package in a page load it is not possible. Just posting it here that we have to do something to improve the situation.
@laoneo Since you have access to an affected server, can you please try something really easy for me to know if the solution I have in mind would work?
Download kickstart-pro-7.1.3-dev202209150549-rev2c43bbf.zip and extract it. Rename the kickstart.php
file to test123.php
and upload it to an affected server.
Run it and click on Import from URL. Use the URL https://update.joomla.org/releases/4.2.2/Joomla_4.2.2-Stable-Update_Package.zip
and tell me what happens.
I'v started at the same time a wget on the same server of the same url and it takes ages to download.
OK, this is expected.
The URL we gave it (https://update.joomla.org/releases/4.2.2/Joomla_4.2.2-Stable-Update_Package.zip) is actually a redirection. Therefore doing a HEAD on it does not return a file size to let us update the percentage. I can address it in the PR by following redirections before doing the HEAD request.
That it worked at all is great. It means that the affected server doesn't time out with 1MiB chunks. So, we can actually provide a workaround.
I will make a PR later. I am currently busy with debugging other stuff.
Thank you very much!!
@laoneo how long does the download take from https://update.joomla.org/releases/4.2.2/Joomla_4.2.2-Stable-Update_Package.zip ?
Last time I measured, it was over 7 minutes.
from update.joomla.org? That makes no sense because update.joomla.org is behind cloudflare and has nothing todo with s3
@HLeithner Let me blow your brain even more. When he tried chunked download (downloading 1MiB at a time, using a Range header) it worked much faster than wget
trying to get the entire file. I think there's some network handling shenanigans going on with this host.
seems that swiss is broken... if you go this road can we please get a progressbar for the download... at the moment it's really a pain if your host needs 30 seconds for the 1st step and you forget if you clicked "update now"
@HLeithner That's the idea, yes. There will be a progress bar.
Thanks, we should add a FAQ article for this issue too.
Have not checked myself yet but we should see where we used timeouts already.