This is not a bug. We use cURL and no issues. If a server has no curl installed than the the server must have allow_url_fopen = enabled. The HTTP/Stream requires PHP's fopen() function and allow_url_fopen enabled

Hi Leo,

euh no. If you use Curl, then you don't need allow_url_fopen. That's the whole point.

And just to make it clear this has always been a requirement since the utility was introduced and is nothing new to Joomla 3.7 nor related with the release

Hi

And just to make it clear this has always been a requirement

It's new, there was no problem updating all our sites from 3.0 and up, with that PHP setup.
This appeared when trying to update to 3.7, because the change was introduced in 3.6.5.
It will occur from now on, because the same code is still used on 3.7.

Have you looked at the code? it's pretty straightforward. The change was made here: 7aa5e65, and included in J! 3.6.5

is nothing new to Joomla 3.7 nor related with the release

It is related in 3.7 in that 3.7 is the first release after 3.6.5, where the issue was introduced.

If allow_url_fopen is now a Joomla requirement, it should be marked pretty clearly, and that's also a B/C break.

Relying on Curl, and falling back on streams, in my experience, is simpler, more efficient and reliable. This is exactly what the JHttpClient does, and there is no reason to not use it that I can see.

Rgds

@weeblrpress Correct re. curl and I have modified my initial reply with better wording (sorry I am not native) I am fully aware that the Joomla Update component will, in case "allow_url_fopen" is disabled, fall back on using cURL wrapper (given cURL PHP extension is enabled). However is the PHP-extension is not installed and allow_url_fopen is disabled you will have no way to get the updater working imho and that has never been the case

Also you might like to read an article from July 2015 #7369 which simply addresses this very same issue so again it is not since Joomla 3.7 and we have that (needs an update though) https://docs.joomla.org/Security_Checklist/Hosting_and_Server_Setup

Use allow_url_fopen
This option enables the URL-aware fopen wrappers that enable accessing URL object like files. Default wrappers are provided for the access of remote files using the ftp or http protocol, some extensions like zlib may register additional wrappers. Note: This can only be set in php.ini due to security reasons. Enable and use allow_url_fopen to allow Joomla's One-Click-Update to work properly.

Hi Leo,

indeed there is some confusion here. Let me try to put things in order:

1. Documentation link: it does suggest to have allow_url_fopen enabled, but it does not say that any update is now impossible if it is disabled

the Joomla Update component will, in case "allow_url_fopen" is disabled, fall back on using cURL wrapper (given cURL PHP extension is enabled)

Actually it's the opposite. The downloading of the file is first attempted with Curl, then streams is used as a fallback (see this line of the model: https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_joomlaupdate/models/default.php#L325 )

However is the PHP-extension is not installed and allow_url_fopen is disabled you will have no way to get the updater working imho

This IS the bug I am reporting. As mentioned above, the Curl extension is installed, enabled and working fine with Joomla (which uses it for new version detection for instance) or other extensions. If cuR

The issue introduced in Joomla 3.6.5 which I described above incorrectly uses the php get_headers() function which requires allow_url_fopen, and does not fallback on Curl or on anything else.

Joomla has an HTTP client, which is used throughout Joomla, as a wrapper meant to allow remote HTTP fetch independantly of the server setup. It should be used as often as possible, but for this particular change, it was not, and instead get_headers was used, thus blocking the cUrl fallback.

I don't think it 's a good idea to require allow_url_fopen, it's a B/C break anyway. Using the official Joomla HTTP client is the way to go, and it preserves the ability to perform updates on servers with cUrl but without allow_url_fopen.

Rgds

@weeblr The main thing needed here was to save the downloaded file under the name without the ? because ? Is a forbidden character for filenames on Windows.

We can use JHttp to get the correct response headers, I would think, the JHttpResponse object includes a headers array and it is already internally following redirects. What we might not be able to get out of JHttp though if it's not included in the response data, which may be why get_headers() was used, is if the request was redirected what the redirected URL is. But that can even be worked around, tell JHttp to not follow redirects and extract the needed info manually.

Either way the code that was added is valid, we just need to improve it a bit to use our abstraction layer versus something that creates a hard dependency.

Hi

@mbabker

What we might not be able to get out of JHttp though if it's not included in the response data, which may be why get_headers() was used, is if the request was redirected what the redirected URL is.

Yes, pretty sure it would be hard to get the follow_location target, as how it's followed is transport-dependent I think.

But that can even be worked around, tell JHttp to not follow redirects and extract the needed info manually.

That's an option, but I have to look at the code: if JHttp makes the redirect target availabe in some way, then we can already solve the problem by adjusting the name during the download phase, not where it's done now. But please read further, I actually don't understand why we need to follow redirects to modify the file name.

@wojsmol Thanks for pointing that out, I do remember now. What I don't understand however is why we have to follow the redirect(s) to modify the file name?
If the original URL has a ?, then strip if and use that as a file name, not sure why we need the final URL instead of the original URL?

Maybe we need to hear from @milux and @rdeutz to understand better the intent?

At this point, my suggestion would simply be to remove entirely that "follow redirects" part, and just sanitize the file name if it has a ?
Or we could just base64 encode the URL, use that as a file name, and don't be bothered by the question marks?

I see no reason to do that only on the final URL. The filename compted at this stage is returned and used throughout, so there is no problem modifying it as we see fit.
But there might be a reason, so would be great to hear from others...

Rgds

Actually I can see now that if the original URL is fixed, something like domain.com/current_version.tar.gz and the download site internally redirects that to domain.com/version_1.2.3.tar.gz,w e have a problem: as the file name is always the same, we won't download a new version, as we think it's already there. Meaning we have to always re-download the file, even if it's there.

So maybe working JHttp client is the way to go, if that's even possible.

wasnt the follow redirects stuff there because of the way that github stores and handles the files

I would suspect so, but the question is more why should we use the final URL (ie possibly invalid because of the ? in it) to store the downloaded file I think.

That I dont remember - I do remember that it was JCE that first reported this type of issue

Pending more details from Robert or other involved, it's probably safe to assume there was a reason, and so I'll look into how the same can be achieved without that hard requirement on allow_url_fopen.
If not possible, then maybe allow_url_fopen should become a requirement.

A quick test on JHttp shows that the final target redirect is not available in the object returned.
So maybe get_headers is indeed the only solution...

You would have to go through com_joomlaupdate and look at where it references a filename to figure out if it's really dependent on the "right" name. Otherwise, you need to be careful that parsing a URL and simply taking the last path segment and stripping the query string still gives a valid filename with file extension. And I think our internals when processing packages (both the update system and the regular extension library) require a valid filename with valid extension to operate correctly (so you couldn't just tell it to unpack foo, it has to be foo.zip).

You would have to go through com_joomlaupdate and look at where it references a filename to figure out if it's really dependent on the "right" name.

From my understanding, com_joomlaupdate is not the issue here, it's more to do with the origin server for the downloads. I still don't really understand the problem, so it's hard to go any further. I did check com_joomlaupdate, and the only thing it needs is a simple (and of course valid) filename.

Otherwise, you need to be careful that parsing a URL and simply taking the last path segment and stripping the query string still gives a valid filename with file extension.

And yet this is what we do ATM.

Going to try dig out how this issue with ? in file names came about. Currently, the filenames listed in manifests are straifghtforward and not generic, ie:
https://downloads.joomla.org/cms/joomla3/3-7-0/Joomla_3.7.0-Stable-Update_Package.zip
so as @brianteeman hinted at, it must have to do with how the files are hosted, but even that I don't understand how it can have an impact on the actual file name.

The downloads site uses Akeeba Release System. Its routing by default doesn't support the URL that you posted, I hacked that support into the router code after the original issue came up and we changed from using ARS default URLs to the "real" filenames again (see https://github.com/joomla/update.joomla.org/pull/30/files for the update server change). How the files are hosted (FWIW they're on S3) is really unimportant at this point other than the fact that there is a redirect issued when requesting the file package for obvious reasons.

Even though we use that URL structure, changing the URL structure in the update server doesn't fix the core issue of trying to write to the filesystem based on the last path segment of the URL and including the query string which uses an invalid character for filenames on Windows (the question mark).

I see. And indeed, ARS append a query at the end of each download URL...

But you see this is what I don't understand so far: in the manifest, we have a perfectly valid URL with a valid file name, without any ? in it. So we could use that directly and not bother about following the redirects that you have to do to work with ATS.

I'm still puzzled as to why we need to figure out the final redirect target and use that to derive a file name, while we could just use the original name??

The core code change was made independent of me hacking ARS.

com_joomlaupdate has a feature for uploading packages and bypassing the download. For that to work though, the uploaded filename has to match the offered download filename. So if user uploaded Joomla_3.7.0-Stable-Update_Package.zip without the ARS change or the change to follow headers it would try to look for joomla_3-7-0-stable-update_package-zip?format=zip&jcompat=my.zip which obviously can't work on Windows. So in the update component it's actually a bit more important that the package names match up.

Well yes. Exactly. The ARS changes are done now, so we should always use "Joomla_3.7.0-Stable-Update_Package.zip"

Where would the "joomla_3-7-0-stable-update_package-zip?format=zip&jcompat=my.zip" come from in that scenario? either when doing an upload or using the standard download method?

If you feel like 7aa5e65 needs to be reverted feel free to propose a PR doing so. But that's the history of why it was needed, and because the release team at the time decided it wasn't urgent enough to push a separate download for the update component to fix the ability for Windows systems to update with the default ARS URLs, I had to hack ARS to work around that decision and keep users on Windows platforms able to do core updates. Right or wrong the code is there now.

My understanding of the events is:

• using ARS, so public URLs have a ? in them
• changes introduced in J! to follow redirects to find final name, because ARS does a redirect from the its "public URL" to the actual location of the file. As the public URL has a ?, it can't be used directly to identify the download. Following the redirect allows finding out the real filename
• then you hack ARS, and we are now back to having public URLs with a valid file name in them
• but that change (which makes allow_url_fopen mandatory) is already in

If this is correct, and there isn't any other release and infrastructure consideration that I have no idea of, then reverting the change would indeed restore ability to update Joomla without allow_url_fopen .

I won't do a PR to revert the changes that without a clear view of why it was added, and whether it's safe to remove it. For instance, will we keep using your hack, or find another way, to be sure URls do not have ? in them? Not knowing about releases or infrastructure, I'm not confident enough here.

Did you have a look at #12817?
The problem was that the updater tried to create a file containing characters that were illegal on windows platforms, because the query string was included to obtain a "file name" with a ".zip" file extension.
In order to fix this and obtain a valid filename with a proper extension, I suggested that fix that resolves the "real" name of the archive by following the redirects.
The JFile::exists($target) prevented the use of a constant archive name. Actually, without following redirects it's imho impossible to find whether the same file has already been downloaded from AWS or whatever source.

If there is some abstraction class in Joomla! that provides header information from URLs in a way that doesn't cause a hard dependency, it would definitely be a good idea to place it there.
I usually look for such options before trying the good old PHP commands, but I might have overlooked something here, so please enlighten me! 😄

EDIT: I want to further clarify my problem: There is JHttp, yes, but how do I get the headers for an URL without starting a real download? That would be a HUGE waste of time and resources...

What just came to my mind:
What is the reason for the existence check JFile::exists($target) in the first place?
The updater checks for a new version in the first step anyway, right?

So, couldn't we use a constant file name and remove all redirection follow code and existence check stuff, just downloading a fresh version every time?
That would definitely reduce the complexity significantly, and make many (also future) problems disappear...

Did I miss something crucial here?

A constant file name is out of the question as that would mean overwriting files at each release for some resource. You also would have no way of knowing if package.zip is for 3.2.1 or 3.7.0 without extracting it.

True, that's exactly the problem I was mentioning.
But I don't consider it an option to download the file multiple times either, how about you?

Is there any chance to know the file name before the download starts and passing it via some parameter?
You also discussed that basename() would now be reliable again because the presented path changed, right? How durable is that change? Is just reverting my patch really an option, or will we start over with the issue once again in a couple of months? ^^

The problem was that the updater tried to create a file containing characters that were illegal on windows platforms, because the query string was included to obtain a "file name" with a ".zip" file extension.
Yes but this only happened because, due to using ARS, the downloaded file was not a real file name, but instead a URL such as joomla_3-7-0-stable-update_package-zip?format=zip&jcompat=my.zip

As Michael explained, this was solved some times ago, and now all download files name are real file names, without any ? in them (ie Joomla_3.7.0-Stable-Update_Package.zip).

Meaning we don't have to worry about ? per se (I would still leave the check for invalid characters in, just in case, but that's another discussion).

In other words, the reason to try to follow redirects has disappeared.

Is there any chance to know the file name before the download starts and passing it via some parameter?

No reason to do that, as the file name is now valid.

How durable is that change? Is just reverting my patch really an option, or will we start over with the issue once again in a couple of months? ^^

That is a valid question. Can the rule of presenting for download a valid file name be set in stone for future releases?

Now if you look at the update manifest file format, it may be possible to also provide a valid file name though:

<update>
<name>Joomla! 3.7</name>
<description>Joomla! 3.7 CMS</description>
<element>joomla</element>
<type>file</type>
<version>3.7.0</version>
<infourl title="Joomla!">
https://www.joomla.org/announcements/release-news/5703-joomla-3-7-is-here.html
</infourl>
<downloads>

<downloadname>
Joomla_3.7.0-Stable-Update_Package.zip
</downloadname>

<downloadurl type="full" format="zip">
https://downloads.joomla.org/cms/joomla3/3-7-0/Joomla_3.7.0-Stable-Update_Package.zip
</downloadurl>
</downloads>
<tags>
<tag>stable</tag>
</tags>
<maintainer>Joomla! PLT</maintainer>
<maintainerurl>https://www.joomla.org</maintainerurl>
<section>STS</section>
<targetplatform name="joomla" version="3.[2345678]"/>
<php_minimum>5.3.10</php_minimum>
</update>

That filename may then be passed through JFile::makeSafe() just to be sure, and used directly.

Is the downloadname tag content actually independent from the downloadurl tag content, or does this data come from the same source?

This is a suggestion, not yet reality. You mentioned passing the file name as a parameter, this is how it could be done. Obviously, the release team would have to implement that new field ,if deemed a valid solution.

Is this still an issue that needs resolving?

Yes. Unable to update to 3.7.5 at the moment.

My 2 cents about the update problem:

Our previous provider had curl activated and updates were no problem at all. Now we switched to a new one, wich had curl deactivated but allow_url_fopen activated. Sounds like the perfect combination for Joomla.

Problem was, that although allow_url_fopen was enabled, for some reason Joomla always tried to use curl as transport. Seems like the autodetection of Joomla doesn't work 100% if curl_exec() is allowed or not.

My quick dirty fix was to remove the CurlTransport.php from /libraries/src/Http/Transport, which forced Joomla to use StreamTransport.php.

@mbabker you're the "infrastructure guy"

Any thoughts on this. For me it sounds like it might be related to the server issue that Nik discovered and has been fixed and even if its not that and something completely different it obviously isnt affecting anyone else or we would have heard about it by now.

Thank you for raising this issue.

Joomla 3 is now in security only mode with no further bug fixes or new features.

As this issue doesn't relate to Joomla 4 it will now been closed.

If we are mistaken and this does apply to Joomla 4 please open a new issue (and reference this one if you wish) with updated details for testing in Joomla 4.
cc @zero-24