? ? ? Pending

User tests: Successful: Unsuccessful:

avatar laoneo
laoneo
19 Jan 2022

Summary of Changes

Downloading the Joomla update package runs on some web sites into a timeout for the main Joomla download server and crashes the update. This should be catched and then the fallback download servers used instead.

The timeout is set to 30 seconds. Not sure if this is too short on slower hostings. Any recommendations?

[UPDATE]: As this here is more of a hot fix. End goal should be to fetch the update package in an ajax request as came out in the discussion of this pr.

avatar laoneo laoneo - open - 19 Jan 2022
avatar laoneo laoneo - change - 19 Jan 2022
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 19 Jan 2022
Category Administration com_joomlaupdate
avatar laoneo laoneo - change - 19 Jan 2022
The description was changed
avatar laoneo laoneo - edited - 19 Jan 2022
avatar zero-24
zero-24 - comment - 19 Jan 2022

Thanks, we should add a FAQ article for this issue too.

Have not checked myself yet but we should see where we used timeouts already.

avatar brianteeman
brianteeman - comment - 19 Jan 2022

This should be catched and then the fallback download servers used instead.

Dont know what they are but probably should also display a message with advice how to manually download the update and then update from that file

avatar zero-24
zero-24 - comment - 19 Jan 2022

The intention is to fallback to github and update.joomla.org first but yes on a follow up PR we could add a message when all three sources failed to provide the update.

avatar HLeithner
HLeithner - comment - 19 Jan 2022

sadly aws is not really fast to all isps, I have servers seen which downloads only with 153KB/s from aws US which means 2m 46s for download. (github mirror needs 8,9 seconds, update.joomla.org needs 5.1s)

Since the timeout parameter implements a hard timeout (complete operations have to be completed within 30 seconds) means the connection needs to have a minimum bandwidth of 9MBit. That doesn't sound much for most countries, but looking at the server mentioned it's located in Austria at Magenta (daughter company of T-Mobile Germany).

So the right value should be based on the minimum bandwidth we would allow to download the update.

Better solution would be to complete the change from AWS as primary download source to update.joomla.org (which is behind cloudflare) and drop (our own AWS) mirror. At least that should make it better for countries outside the US.

Edit: Changed the cloudflare part

avatar zero-24
zero-24 - comment - 19 Jan 2022

So 30 seconds would work here as with this PR we would fallback to github and update.joomla.org which are well below the 30 seconds we set as limit or i'm missing something?

An change to cloudflare can and in my understanding is already beeing activly worked on but should not prevent us from patching this issue here right?

avatar webuniverse
webuniverse - comment - 19 Jan 2022

I have tested this item successfully on aaf3111


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36741.

avatar webuniverse webuniverse - test_item - 19 Jan 2022 - Tested successfully
avatar Fedik
Fedik - comment - 19 Jan 2022

In my opinion 30sec is very small, especially for joomla 4 where package much bigger in size.
Maybe something around 3-5min at least.

avatar brianteeman
brianteeman - comment - 19 Jan 2022

are you measuring the time to start the download or the time to complete the download

avatar Fedik
Fedik - comment - 19 Jan 2022

the time to complete the download

avatar zero-24
zero-24 - comment - 19 Jan 2022

In my opinion 30sec is very small, especially for joomla 4 where package much bigger in size.
Maybe something around 3-5min at least.

Yes I get the point but the problem is the maximal timeout of the server right? The timeout we are faceing where the webserver of the site times out (because AWS took to long)

avatar laoneo
laoneo - comment - 19 Jan 2022

are you measuring the time to start the download or the time to complete the download

To establish the connection and to complete it:

avatar blueforce
blueforce - comment - 19 Jan 2022

I have tested this item successfully on aaf3111

Tested with php8...


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36741.

avatar blueforce blueforce - test_item - 19 Jan 2022 - Tested successfully
avatar brianteeman
brianteeman - comment - 19 Jan 2022

@laoneo but that says if there is no time out set then

the Default timeout is 0 (zero) which means it never times out during transfer.

avatar laoneo
laoneo - comment - 19 Jan 2022

And this is what exactly happens. So the download takes ever and on some point the max execution time of the web server, where Joomla is running on, kills the process with a timeout error. It is an issue between your web browser and the Joomla server and not between the Joomla server and the download server from joomla.org. As if the web server would allow a max execution time of, lets say an hour, then the update will succeed.

avatar richard67 richard67 - change - 19 Jan 2022
Status Pending Ready to Commit
Labels Added: ?
avatar richard67
richard67 - comment - 19 Jan 2022

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36741.

avatar brianteeman
brianteeman - comment - 19 Jan 2022

seems very wrong to me

avatar zero-24
zero-24 - comment - 19 Jan 2022

seems very wrong to me

What seems very wrong to you? We have two different times to cosider the time it takes to download from AWS and the time our webserver allows us to run. Right now we dont limit the time to download so the webserver kills the process. With this patch we limit it ourself so we can fallback to other locations when AWS is not as fast as required.

avatar HLeithner
HLeithner - comment - 19 Jan 2022

The approach is to a bit too simple and doesn't consider default php max_execution_time which is per default 30 seconds, Webhosting company may change this to a higher value.

The problem is that you may help websites getting a timeout because they have only 60 seconds to download but kill webserver hat needs 70 seconds to download....

I would suggest a combination of 3 values (most of them only work with curl I think and may nott exposed to the joomla http transport shim).
set CURLOPT_CONNECTTIMEOUT to 5 seconds
set CURLOPT_LOW_SPEED_TIME to 5 seconds
set CURLOPT_LOW_SPEED_LIMIT to 10 Mbit/s

Setting the minium speed to 10 Mbit/s means that Joomla needs about 24 Seconds to download. if the connection rate is slower then this we would switch to the next fallback host which hopefully is faster then the first one and we still have enough time to download the package. See my example above, 2 minutues for AWS-US and 9 seconds for github and 5 for upload.joomla.org

Of source this would force a internet connection from minimum of 10 Mbit/s

avatar brianteeman
brianteeman - comment - 19 Jan 2022

Finally a voice of sense.

Setting the minium speed to 10 Mbit/s means that Joomla needs about 24 Seconds to download.

Have you tried to update joomla on a local development machine on a 256k connection?

avatar brianteeman
brianteeman - comment - 19 Jan 2022

What seems very wrong to you? We have two different times to cosider the time it takes to download from AWS and the time our webserver allows us to run. Right now we dont limit the time to download so the webserver kills the process. With this patch we limit it ourself so we can fallback to other locations when AWS is not as fast as required.

You are not considering the connection speed

avatar zero-24
zero-24 - comment - 19 Jan 2022

What seems very wrong to you? We have two different times to cosider the time it takes to download from AWS and the time our webserver allows us to run. Right now we dont limit the time to download so the webserver kills the process. With this patch we limit it ourself so we can fallback to other locations when AWS is not as fast as required.

You are not considering the connection speed

What connection speed? Is the connection speed from me to my server actually relevant? The only thing thats send between me and my server is the html to render the Joomla Update site right? The update itself will be downloaded by the server without any data send to my local pc or i'm missing something?

avatar brianteeman
brianteeman - comment - 19 Jan 2022

... and if your server is local?

avatar zero-24
zero-24 - comment - 19 Jan 2022

... and if your server is local?

Ok good that i asked i hadn't that in mind. Hmm so we actually have to check the max_excecution time too? Whe its set to 0 (infinit) its easy but i dont know how we could best do it for the other settings. Cause we cant just take whats set there as we need a bit more time after the download.

avatar brianteeman
brianteeman - comment - 19 Jan 2022

finally

avatar brianteeman
brianteeman - comment - 19 Jan 2022

That's why I asked you

are you measuring the time to start the download or the time to complete the download

avatar richard67
richard67 - comment - 19 Jan 2022

@zero-24 Shall I remove RTC for now?

avatar brianteeman
brianteeman - comment - 19 Jan 2022

This should be catched and then the fallback download servers used instead.

Dont know what they are but probably should also display a message with advice how to manually download the update and then update from that file

All you can do is to display the message. If the download takes more than your arbitrarily defined time to download from one server it will almost certainly time out on any other server.

You need to go back to first principles. What is the problem I am trying to solve. Do I know anyone with experience in this specific area that can help/advise/guide

avatar Fedik
Fedik - comment - 19 Jan 2022

All you can do is to display the message

I think this is another side of problem, the downloads start on "form submit", or I am wrong?
And if User will get Apache timeout, you cannot catch this error and display correct message. User will see white page or page with default server error.

The process could be like this:

  • User click update
  • Updater redirects to the update page, where run AJAX request to download (maybe even with download progress)
  • Here we could catch local server timeout or server-to-server download error and show an error message
avatar laoneo
laoneo - comment - 20 Jan 2022

So to move on here I propose to merge it as a quick fix and then open an issue which will address a new feature regarding the download in an ajax request.

avatar laoneo laoneo - change - 20 Jan 2022
The description was changed
avatar laoneo laoneo - edited - 20 Jan 2022
avatar brianteeman
brianteeman - comment - 20 Jan 2022

Please dont. You are breaking more than you are fix

avatar laoneo
laoneo - comment - 20 Jan 2022

What is broken?

avatar brianteeman
brianteeman - comment - 20 Jan 2022

Already stated.

avatar laoneo
laoneo - comment - 20 Jan 2022

If the download takes more than your arbitrarily defined time to download from one server it will almost certainly time out on any other server.

As you can see for the actual problem this was not the case. The download to AWS took ages and the one to github fast fast.

This pr limits the time to fetch the package and when all (including the fallback) are failing then the checksum check here should return false . Which will lead to an exception as you can see here that the checksum is wrong. So actually the behavior is correct.

What can be discussed is if the timeout of 30 seconds is too restrict.

avatar brianteeman
brianteeman - comment - 20 Jan 2022

Please provide test instructions to validate that after 30 seconds instead of crashing the download is switched from aws.

avatar Fedik
Fedik - comment - 20 Jan 2022

Please provide test instructions

With assumption you use local server, I think it can be tested like next.
Temporary edit "hosts" file of your OS to redirect downloads.joomla.org to localhost:

127.0.0.1 downloads.joomla.org

Or block this domain in your router.

Then try update (probably need some dev channel, not sure here, or install older version first).
It should wait 30 sec, then switch to one of:

github.com/joomla/joomla-cms/releases/download
update.joomla.org/releases

And finish the update.

avatar brianteeman
brianteeman - comment - 20 Jan 2022

and without this pr you are saying that it will never switch to one of the other servers?

If so then I can not replicate the original problem so cannot test this "fix"

How have the two testers actually tested that this works.

avatar zero-24
zero-24 - comment - 20 Jan 2022

Because the actual problem is not that downloads is not reachable. Downloads is reachable the underling US AWS is slow on some hosts as described above by Harald. So the test would be requiringing to have an affected host.

Or simulating an reachable but very slow replying AWS.

avatar brianteeman
brianteeman - comment - 20 Jan 2022

well I tried simulating it using network throttling and did not have any issues even when throttling down to a 50kbs connection

avatar laoneo
laoneo - comment - 20 Jan 2022

How and more important, where, did you set network throttling?

avatar brianteeman
brianteeman - comment - 20 Jan 2022

I tried it in two places.

  1. Chrome browser
  2. browserstack
avatar zero-24
zero-24 - comment - 20 Jan 2022

Chrome browser

Thats the thing that does not have an affect here as its between the client and the server. The issue here is between your server (even when that is localhost) and aws where crhome does not have any impact.

browserstack

Dont know the details how browserstack works but from its name it does not sound like that you acutally limit the bandwith that the hosting server has aviable to download the package.

The other way arround could be a changed update server where you point to a never responding server I'm not sure how such thing would have to be setup.

But i understand the point that there could be issues for localhost sites not getting the package within 30 seconds but they have very long max_execution times (even 0=infinite) when the local PC (which would be the server in that case) not get any of the three packages within the 30 seconds the update fails.

The aviable max_execution time has also be devided by three given that we try three different locations too right? Could that be a workaround? What about checking that time and try to set that? And for example do not limit it when its set to 0? What happens when AWS does not come back at all (or runs infinite) at somepoint we should step up and direct the users to a different mirror right?

avatar Fedik
Fedik - comment - 20 Jan 2022

The other way arround could be a changed update server where you point to a never responding server I'm not sure how such thing would have to be setup.

See my previous comment ?
Can as well, use non existing IP like 196.100.100.100 for hosts

avatar laoneo
laoneo - comment - 20 Jan 2022

Ok @brianteeman, you still didn't understand the issue here. Follow the steps and then you will see the timeout:

It will timeout as the update is fetched from https://joomla.digital-peak.com/slow.php?file.zip where I do a sleep of 99999 seconds. When you apply the patch, then the update will work. For Joomla 4 you need the patch from #36742.

avatar brianteeman
brianteeman - comment - 20 Jan 2022

Clearly not just me as others have said to do X to test which was not correct

avatar laoneo
laoneo - comment - 20 Jan 2022

And could you reproduce the issue?

avatar brianteeman
brianteeman - comment - 20 Jan 2022

Sorry I was watching the football.

Trying to follow your instructions

  1. done
  2. done
  3. done
  4. Check that in the table j_updates the entry with the id 42 has https://joomla.digital-peak.com/joomla.xml as details url
    There is no entry in this table that matches. There are only language update sites. Did you mean j_update_sites and the location url?
  5. done
  6. no update found
avatar brianteeman
brianteeman - comment - 20 Jan 2022

trying again on a new server

avatar brianteeman
brianteeman - comment - 20 Jan 2022

same as before with point 4 but now update found in point 6

Try to update - eventually get a 500 error

applied patch from 36742

Try to update - eventually get a 500 error

Server info
max_execution_time 60
php 8.0.10

avatar laoneo
laoneo - comment - 21 Jan 2022

Sorry I was watching the football.

4. Check that in the table j_updates the entry with the id 42 has https://joomla.digital-peak.com/joomla.xml as details url
   There is no entry in this table that matches. There are only language update sites. Did you mean j_update_sites and the location url?

Didn't know that football is still a thing in England ?

Did you check it with PHPmyAdmin? If yes, then you probably need to got to the second page as it is the last entry in the table. And make sure you have installed version 4.0.5.

avatar brianteeman
brianteeman - comment - 21 Jan 2022

Sorry - I must have been distracted by the football. The record is present. Just retested everything again on a different net connection and still just get the 500

avatar laoneo
laoneo - comment - 21 Jan 2022

Where the patch is applied, can you echo there the url and die. What is first tried for a connection?

avatar nikosdion
nikosdion - comment - 21 Jan 2022

Ideally, Joomla would implement a staggered file download just like I implemented since 7 years ago in FOF 3: https://github.com/akeeba/fof/blob/fof-3.x/fof/Download/Download.php#L263

This can be used to download a file one MB at a time and reassemble it at the server without ever timing out, see https://github.com/akeeba/kickstart/blob/development/source/urlimport/feature.urlimport.php for the code I am using in Kicsktart Professional to import several Gigabyte big files since tern or eleven years ago.

All these problems are solved years ago.

However, here's an interesting thing to think about.

Staggered downloads only make sense when using the web interface. When using the CLI you should use the current method. Therefore you would need to different extension update paths which is impossible in the current architecture. And if you are going to go through this pain, why not implement the whitepaper I had written back in 2012 about making extension updates through the web interface resilient against slow hosts? That's something for @HLeithner to think about as any major change in that are would come with Joomla 5.

avatar brianteeman
brianteeman - comment - 21 Jan 2022

image

image

Is this what you wanted?

avatar nikosdion
nikosdion - comment - 21 Jan 2022

@HLeithner @bembelimen BTW, if you need me to make a PR for chunk downloads of the Joomla Update package for Joomla 4.1 I can do that. This wouldn't touch extensions updates but it would fix downloading the really big Joomla Update package. Between that and the other two PRs I've made (one merged, one pending) that'd be a pretty major upgrade to Joomla Update's reliability.

If you guys want me to do that please tell me today; I might be able to work on that this weekend.

avatar brianteeman
brianteeman - comment - 21 Jan 2022

This is some of the most important code in joomla. If any pr breaks the update process then we're screwed. Thats why I am being so insistent that it needs to be a verifiable fix

avatar laoneo
laoneo - comment - 21 Jan 2022

No problem. In the file where you applied the patch can you change the line to
$result = HttpFactory::getHttp([], ['curl', 'stream'])->get($url,[],30);echo $url;die;.

So this should print the url to download after 30 seconds. And without the patch it will never return.

avatar brianteeman
brianteeman - comment - 21 Jan 2022

So this should print the url to download after 30 seconds.

It doesn't - already tried it but did it again just to be certain

And without the patch it will never return.

It is the patch ??

avatar laoneo
laoneo - comment - 21 Jan 2022

Do you have curl installed? Looks like the timeout doesn't work for some unknown reasons. I'm lost then.

avatar brianteeman
brianteeman - comment - 21 Jan 2022

image

avatar HLeithner
HLeithner - comment - 21 Jan 2022

@HLeithner @bembelimen BTW, if you need me to make a PR for chunk downloads of the Joomla Update package for Joomla 4.1 I can do that. This wouldn't touch extensions updates but it would fix downloading the really big Joomla Update package. Between that and the other two PRs I've made (one merged, one pending) that'd be a pretty major upgrade to Joomla Update's reliability.

If you guys want me to do that please tell me today; I might be able to work on that this weekend.

of course that would make much sense and the upgrade process much user friendly, but I think the target would be 4.2, for 4.1 it's too late, so no hurry.

@roland-d and @fancyFranci is responsible for 4.2 just for tagging this.

avatar HLeithner
HLeithner - comment - 21 Jan 2022

@brianteeman use a local proxy with throttling

  • download and install charles proxy
  • start it
  • if you are on windows goto top menu "proxy -> disable Windows proxy" (that's the default proxy for all browsers and programs, if used)
  • open from the top menu "proxy -> SSL Proxying Settings -> Enable it and click on "add" on the left site under the list ("Include") -> and press OK
  • enable throttling from the top menu "proxy -> Start throttling"
  • edit the file 'libraries/src/Http/Transport/CurlTransport.php' in line 199 (before curl_setopt_array($ch, $options);) and add this line: $options[CURLOPT_PROXY] = '127.0.0.1'; $options[CURLOPT_PROXYPORT] = '8888'; $options[CURLOPT_SSL_VERIFYPEER] = 0;

now your local joomla installation should use the proxy to download files from the internet, for better overview what gets downloaded use the "Sequence"-tab in Charles proxy. You can also change the download speed in top menu "proxy->throttle settings"

avatar nikosdion
nikosdion - comment - 21 Jan 2022

@HLeithner Agreed. I will work on the PR over the next few days. I am now moving out of your way here.

avatar brianteeman
brianteeman - comment - 21 Jan 2022

Thanks @HLeithner

Using your instructions and throttling as I was able to confirm that without this PR it would result in a 500 error when the local server max_execution time was reached.

After applying the patch I can now confirm that after 30seconds the updater switched to the secondary source of github.

So I am happy to assume* that if I reconfigure charles to only throttle amazon it will indeed fallover to github and complete

I say assume because I didnt manage to get charles to only throttle aws

avatar laoneo
laoneo - comment - 21 Jan 2022

Nice you got it sorted somehow, then I delete my public files.

avatar richard67
richard67 - comment - 21 Jan 2022

@bembelimen Can the „Updates requested“ label be removed now as things have been discussed above?

avatar HLeithner
HLeithner - comment - 22 Jan 2022

Sorry to say but I'm not convinced that this is the right way, because people in countries with slower uplink but longer execution time get lockout from the this update process.

avatar brianteeman
brianteeman - comment - 22 Jan 2022

Thats what I started off saying

avatar brianteeman
brianteeman - comment - 22 Jan 2022

also if the server has a max-execution time of 60 then with this code you can never fall back to the third source

avatar laoneo
laoneo - comment - 22 Jan 2022

Sorry to say but I'm not convinced that this is the right way, because people in countries with slower uplink but longer execution time get lockout from the this update process.

It is easy to say that, better would be to propose an alternative.

avatar HLeithner
HLeithner - comment - 22 Jan 2022

@laoneo

It is easy to say that, better would be to propose an alternative.

I did?

The approach is to a bit too simple and doesn't consider default php max_execution_time which is per default 30 seconds, Webhosting company may change this to a higher value.

The problem is that you may help websites getting a timeout because they have only 60 seconds to download but kill webserver hat needs 70 seconds to download....

I would suggest a combination of 3 values (most of them only work with curl I think and may nott exposed to the joomla http transport shim). set CURLOPT_CONNECTTIMEOUT to 5 seconds set CURLOPT_LOW_SPEED_TIME to 5 seconds set CURLOPT_LOW_SPEED_LIMIT to 10 Mbit/s

Setting the minium speed to 10 Mbit/s means that Joomla needs about 24 Seconds to download. if the connection rate is slower then this we would switch to the next fallback host which hopefully is faster then the first one and we still have enough time to download the package. See my example above, 2 minutues for AWS-US and 9 seconds for github and 5 for upload.joomla.org

Of source this would force a internet connection from minimum of 10 Mbit/s

avatar laoneo
laoneo - comment - 22 Jan 2022

Then make a pr and I close this one. I mean it fulfilled it's purpose, it saved Switzerland.

avatar nikosdion
nikosdion - comment - 22 Jan 2022

Hey Allon, I did propose a better solution and I’ll make the PR.

As I said, the problem you’re trying to solve is a solved problem and what you are doing is not the right way to deal with it. It’s not even a reasonable workaround. It’s a bad hack which will cause more problems than it solves.

Right now we have a minority of users who have a problem downloading updates. They can always download the update manually and use the upload and upgrade feature I added back in Joomla 3.4.

What you are doing will lock out many live sites and pretty much ALL locally hosted sites from upgrading with a single click EVEN THOUGH this worked for them just fine before.

Breaking something for at least 100x the number of people you’re fixing it for is NOT a solution. This is mass distributed software. People rely on our work. We can’t go around and break stuff just to serve our own narrow use case, consequences be damned.

The point you raised is valid and needs addressing. This PR is not the way to address it. I will make a PR once I feel better. Wait for a week or less. Don’t break things in the meantime.

I tried saying that politely before but since you seemed to ignore me I have to be more forceful. I don’t like doing this. It’s upsetting to me too.

avatar laoneo laoneo - change - 22 Jan 2022
Status Ready to Commit Closed
Closed_Date 0000-00-00 00:00:00 2022-01-22 12:35:34
Closed_By laoneo
Labels Added: ? ?
avatar laoneo laoneo - close - 22 Jan 2022
avatar laoneo
laoneo - comment - 22 Jan 2022

I do not have a problem to close as it was indeed a hack. But we needed fast a solution and this was one.

avatar laoneo
laoneo - comment - 14 Sep 2022

This is again an issue on some Swiss hosters. We need to find a solution for this!!

avatar brianteeman
brianteeman - comment - 14 Sep 2022

/me wonders why only switzerland

avatar nikosdion
nikosdion - comment - 14 Sep 2022

@laoneo @HLeithner I have talked to both of you about it. This is ultimately NOT something to fix with code, this is an infrastructure issue.

If you want me to implement chunked download for the updates I can do the PR. Do note that if the problem is as bad as Allon and Harald say — a few hundreds Kilobits per second — even that might not be enough as 1MiB (the lowest chunk we can go for without causing network issues…) would take several seconds. Also, I would only make this an optional feature. For people who can use the regular one–shot download it's actually better to stick with it!

Right now, Joomla is trying to use either a download from an Amazon S3 bucket or through GitHub releases (which also uses an Amazon S3 bucket).

The problem is that Amazon S3 buckets are localised, they only exist in the [region](https://docs.aws.amazon.com/general/latest/gr/s3.html_ they are created in. For example, an S3 bucket in the eu-north region only exists in a few datacenters around Stockholm. If you try to reach that from Tokyo or, worse, from the Pacific the data transfer speed will suck incredibly and appear almost stalled.

At the very least, Joomla should use multi-region access for its bucket or use an Amazon CloudFront CDN. I can't tell you which one would work better for you without having the number of requests and transfer volume but you having all that information can use Amazon's cost calculators to figure it out.

avatar brianteeman
brianteeman - comment - 14 Sep 2022

I thought that we took advantage of an aws grant to cover the costs?

avatar laoneo
laoneo - comment - 15 Sep 2022

@nikosdion despite it is an infrastructure issue, somehow we should be able to handle this, even when it is a simple download speed information or whatever. I know with the current approach, where we download the package in a page load it is not possible. Just posting it here that we have to do something to improve the situation.

avatar nikosdion
nikosdion - comment - 15 Sep 2022

@laoneo Since you have access to an affected server, can you please try something really easy for me to know if the solution I have in mind would work?

Download kickstart-pro-7.1.3-dev202209150549-rev2c43bbf.zip and extract it. Rename the kickstart.php file to test123.php and upload it to an affected server.

Run it and click on Import from URL. Use the URL https://update.joomla.org/releases/4.2.2/Joomla_4.2.2-Stable-Update_Package.zip and tell me what happens.

avatar laoneo
laoneo - comment - 15 Sep 2022

It showed me 0% with 8 mb, then 0% with 9mb and all of the sudden it wen to the following screen:
image

The file is fully downloaded

avatar laoneo
laoneo - comment - 15 Sep 2022

I'v started at the same time a wget on the same server of the same url and it takes ages to download.

avatar nikosdion
nikosdion - comment - 15 Sep 2022

OK, this is expected.

The URL we gave it (https://update.joomla.org/releases/4.2.2/Joomla_4.2.2-Stable-Update_Package.zip) is actually a redirection. Therefore doing a HEAD on it does not return a file size to let us update the percentage. I can address it in the PR by following redirections before doing the HEAD request.

That it worked at all is great. It means that the affected server doesn't time out with 1MiB chunks. So, we can actually provide a workaround.

I will make a PR later. I am currently busy with debugging other stuff.

avatar laoneo
laoneo - comment - 15 Sep 2022

Thank you very much!!

avatar nikosdion
nikosdion - comment - 15 Sep 2022

@laoneo Just to be sure, I will make the PR against 4.2 as 3.10 is now in maintenance mode, only fixing security issues. Right?

avatar laoneo
laoneo - comment - 15 Sep 2022

I would say yes, as when there is a security release and people can't update because of this issue, then they can at least upload the package manually. But I did ping @zero-24 to make a comment.

avatar HLeithner
HLeithner - comment - 15 Sep 2022
avatar laoneo
laoneo - comment - 15 Sep 2022

Last time I measured, it was over 7 minutes.

avatar HLeithner
HLeithner - comment - 15 Sep 2022

from update.joomla.org? That makes no sense because update.joomla.org is behind cloudflare and has nothing todo with s3

avatar nikosdion
nikosdion - comment - 15 Sep 2022

@HLeithner Let me blow your brain even more. When he tried chunked download (downloading 1MiB at a time, using a Range header) it worked much faster than wget trying to get the entire file. I think there's some network handling shenanigans going on with this host.

avatar HLeithner
HLeithner - comment - 15 Sep 2022

seems that swiss is broken... if you go this road can we please get a progressbar for the download... at the moment it's really a pain if your host needs 30 seconds for the 1st step and you forget if you clicked "update now"

avatar nikosdion
nikosdion - comment - 15 Sep 2022

@HLeithner That's the idea, yes. There will be a progress bar.

avatar nikosdion
nikosdion - comment - 15 Sep 2022

@laoneo It took me longer than I expected as I kept finding new ways to break Joomla Update BUT I now have a PR for you to test: #38774

Add a Comment

Login with GitHub to post a comment