This is expected as allow_url_fopen is required
https://docs.joomla.org/Security_Checklist/Hosting_and_Server_Setup

Further reading http://www.dionysopoulos.me/flicking-the-wrong-switch.html

Your point being?

Perhaps a better reply/suggestion would be:

Is there a way Joomla can detect the allow_url_fopen PHP configuration option and show the user a warning message on the Joomla Update screen?

If I remember correctly there is a warning during the pre-installation
check.

If someone has an code proposal for a warning message in the updater then I
am sure it will be considered.
On 7 Jul 2015 20:44, "justinherrin" notifications@github.com wrote:

Perhaps a better reply/suggestion would be:

Is there a way Joomla can detect the allow_url_fopen PHP configuration
option and show the user a warning message on the Joomla Update screen?

—
Reply to this email directly or view it on GitHub
#7369 (comment).

No, the pre-installation check does not include that option.

Hi everyone,

1. @chrullrich: I tested following your report and it turns out you're only partially correct about the misbehaviour you reported here.

   • on a clean (files/db) J! 3.4.2 install & "allow_url_fopen = 0" - the message New version 3.4.3 pops up - no problems
   • even with cURL extension disabled - the new version notification pops up
   • however, when I proceeded with upgrade to v.3.4.3, and then edited "version.php" file, trying to fool Joomla into thinking it's still on 3.4.2 - only then the situation you described appears. System states it's on v.3.4.2 AND no new version was available. I didn't go investigating why, but it doesn't really matter, that was a forced situation ...

2. I read now and then on forums and chats that allow_url_fopen has to be enabled for Joomla to work properly. That's an "urban legend", leftover from old versions, but almost no one seems to actually test and confirm that fact. It seems the Docs are also wrong about it.
   Joomla! CMS & Framework don't depend on "allow_url_fopen" being enabled. Period!
   Quite the contrary, J! Framework has a very good and robust core subpackage HTTP, which supports "drivers" based on cURL, streams and sockets. Which means even with fopen() URL wrappers ("streams" driver) disabled altogether ("allow_url_fopen = 0"), any extensions implementing HTTP subpackage still has "cURL" and "sockets" driver available. For example, Joomla Update component will, in case "allow_url_fopen" is disabled, fall back on using cURL wrapper (given cURL PHP extension is enabled).

3. And now we come to the crucial part - com_joomlaupdate's author, for whatever odd reason he had, got himself into reinventing the wheel by writing custom fopen() and cURL wrappers. Instead of simply using mentioned HTTP subpack. And, additionally, forgot(?) to implement raw socket communication. Which means if both allow_url_fopen and cURL extension are disabled - Joomla can't download upgrade package.
   But, for example, the Install from Web in the same PHP configuration will still work flawlessly, since it falls back on using HTTP subpack's "sockets" driver.

4. allow_url_fopen is a very controversial parameter. And all that comes from a fact that developers and sysadmins take their highly biased stances based on their own perspectives.

Developers are tending to get the coding job done in the most comfortable way. Writing file_get_contents("http://someunvalidated.com/package.zip") is much easier and "cool" than to implement a proper socket wrapper function. Unfortunately.

(Security aware) sysadmins, on the other hand, try to make everything as airtight as possible. And if they clean as much malware infected websites as I am, on a daily basis, they have noticed that most of the generic backdoor scripts in circulation by script-kiddies, tend to be written in the similar way as mentioned above - implementing some of the fopen() based URL wrappers. Which means that disabling those would probably disarm any automated script-kiddie attack. Which sounds good enough to disable it. And I tend to agree and do the same with my servers, and I can witness it's pretty effective. Regular scans of codebase on serves find backdoors laying around, but no harm done.
So, I understand them both, I am a developer and a sysadmin, providing a lot of security audits and hacked site cleanups. So I'd dare to say I have much broader perspective on the whole "lazy vs. secure" subject.
In the blog post, which @brianteeman linked here, @nikosdion rants about disabled "allow_url_fopen" on some servers, and is being even derogatory about sysadmins disabling it. He tries to convince readers how allow_url_include is a "magic potion" which made URL wrappers benevolent. Come on!
allow_url_fopen enables file_get_contents("http://badsite.com/badware.php")
allow_url_include enables include("http://badsite.com/badware.php")

Now, someone please explain to me why downloading badware.php is GOOD, while in the same time remotely including badware.php is BAD and has to be blocked??
They both enable downloading the same malware. Include() "autoruns it". File_get_contents() stores it on the server and leaves it to happily wait for me to open it up in my browser, and still use it for whatever I wish to...
So, it's just because some developers won't stop using comfortable one-liners and are lazy to find a wrapper functions for cURL or sockets (in Joomla extension case - won't invest some time to explore native packages like HTTP) and implement them.
My point being: please choose, either trust nobody (starting with indifferent hosting server users) and block every wrapper, OR don't worry about any direct remote URL access functions at all, they do almost the same thing - enable downloading (un)trusted remote data!

Quick Solution:

I'll prepare a pull request with quick fix for com_joomlaupdate, to enable usage of native HTTP and socket driver. Which fixes the problem when fopen() URL wrappers and cURL are disabled

Proper Solution:

Someone should rewrite com_joomlaupdate downloader by dropping needles wrapper functions and implementing native HTTP, letting it to decide which driver to use.

Before you get too far into whatever you're planning on doing, there is a very purposeful intent in that some aspects of the update component are isolated from the CMS itself.

Quite frankly, the fact that the component uses CMS functions (and the fact that we are updating Joomla itself as an extension of the CMS; seriously, we're using the file extension type which is why you can install the package through the Extension Manager) in the middle of the update cycle and the update is not fully a fully isolated script is an issue to be addressed IMO.

@mbabker I get what you're getting at, and I agree.
IMO, it would make sense that updater would be a part of CLI

P.S. @mbabker are you suggesting me to write a separate socket wrapper function in com_joomlaupdate, just like other two ?

I think for the actual package download, we would be safe using JHttp as the update hasn't started at that point. But as soon as we start extracting the package and making changes to the filesystem, IMO that would be the point we no longer depend on the core platform. I guess it all depends on the order of operations in the component though.

Yes, that's exactly what I was thinking, since in the phase of download nothing has been updated so far.
https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_joomlaupdate/helpers/download.php

DO NOT USE CORE CODE TO EXTRACT THE PACKAGE. DO NOT SCREW UP JOOMLA! UPDATE TO THE STATE IT WAS BEFORE I FIXED IT IN 2.5.3.

There is a reason we have a separate component and not using the extensions updater. It is called "shared hosting with CPU limits". Do NOT break Joomla because some random person cannot understand the difference between downloading code and running it.

@btoplak Thanks for the analysis work. One thing: Where you say

on a clean (files/db) J! 3.4.2 install & "allow_url_fopen = 1" - the message New version 3.4.3 pops up - no problems

, is that a typo? With the option set to 1, it is enabled, isn't it?

I certainly didn't edit any files to confuse Joomla about which version it was. As far as reproducing the behavior goes, this is what I see:

• Set allow_url_fopen = 0.
• Unpack the 3.4.2 package, follow the installation wizard.
• Result: "Joomla is up-to-date".
• Set allow_url_fopen = 1, kick web server/PHP.
• Reload the control panel
• Result: "Update to 3.4.3 available".

That should be conclusive enough. If not, to double-check:

• Set `allow_url_fopen = 0', kick again.
• Result: Update still reported.
• DELETE FROM whatever_updates;
• Result: "Joomla is up-to-date"

You have the PHP cURL extension installed and enabled. If you disable both fopen wrappers and cURL you will indeed have a broken server which cannot download the update. Solution: fire your host, they do not understand how PHP hosting works (just like the original poster who openly disagrees with the developers of PHP regarding URL fopen vs URL include).

@chrullrich yes, thanks, you are correct about wrong setting quote, that's a c/p typo, I'll fix it now.

I will test again, but the procedure you described is roughly the same I used on my dev EasyPHP machine Apache/2.4 + PHP/5.3

@nikosdion Please read before coming in accusing and SHOUTING. Manners seem to be too much to expect ...

No one said I'll be using J! Framework packages for extracting. We talk about downloading process here. Read.

If both, fopen() URL wrappers and cURL are disabled, there is no dead end, and no server is broken. There is still a huge chance PHP was complied with sockets support enabled (didn't see such odd case of disabled sockets in a loong time). That means any extension implementing JHTML, or implementing sockets, will be able to download without any problems.
Unfortunately, com_joomlaupdater is currently not one of such extensions because some random person didn't remember to implement it.

If you'd have had read, you'd notice that OP didn't talk about url includes, it was me. And they work as I described.

And stop worrying about allow_url_fopen = 0 anymore, Joomla works fine without it, reading my previous post till the end will help understanding why.

@nikosdion This style of discussion makes it difficult to determine which particular statement someone is replying to.

In case you were replying to me, I am the original poster, and I think I understand the central point of this issue quite well. PHP does not separate between code and data, therefore a PHP-enabled web server should never be allowed to access any remote resource whatsoever. Careful whitelisting may work, but, of course, PHP has no way to do that.

Can you please give me some sources that explain why enabling allow_url_fopen is secure? It seems glaringly obvious to me that it is not, and neither are any of its proliferating brethren like cURL, plain sockets, etc.

Sure, it is "secure" when considered in isolation, but when it is finished downloading-not-executing, there is still an exploit sitting on your server's disk, slowly ticking away, waiting for someone to cause the server to run it.

This discussion has already changed direction away from what I originally intended. My report pointed out that the updater flatly lies in the user's face by claiming that an installation is up-to-date when it has no way of determining that. Instead, what is now being discussed is how to use a multi-level fallback hierarchy to try every way of communicating with the outside world that PHP has.

@btoplak I really appreciate your effort, and I'm sure someone else will benefit from it, but no amount of implementing alternative ways of performing outgoing requests from the server will ever fix the problem I reported, because I will always do my best to ensure that no such thing can happen, and that includes disabling access to plain sockets.

The only thing I would consider appropriate for the server to retrieve from outside is the bare information "the most recent release in the 3.x series is 3.4.3", published in some way that does not involve HTTP, FTP, or in fact any socket connection at all in which PHP controls the actual bytes sent or received. DNS might work.

@chrullrich i'll test and try to recreate your original problem again.

But if you want to get trustworthy information - DNS is in no way more trustworthy than HTTP (DNS poisoning, local or remote, etc). My suggestion is HTTPS verified with trusted local rootcert bundle. But, this is really out of topic.

Then again, my comment about Joomla Updater missing socket implementation is offtopic too. I will open a separate ticket and pull request when I get the time to prepare a pull request.

My point was that I would allow the application to determine whether there is an update available, but not what its content is, or the content of any other remote resource.

Also, DNSSEC.

I have to reply from a mobile phone so excuse me for not at mentioning you, OP.

If you can still not understand what is the difference between open and include please RTFA until you do. Do you seriously disagree with the people who wrote PHP about open vs include? I mean, sure, ignore me and my article. Just read the PHP documentation. Where do you think my information comes from? I don't talk to God, I just RTFM.

To spare you some trouble. URL fopen wrappers, cURL and raw sockets can all be used to download data, no matter what it is. Sure you can disable them all, but you are just being a twat because I bet you do have something that supports uploads, eg media manager, which can be theoretically exploited and don't get me started on your server's security. So here is your code upload path. Anyway. According to your idea of security, downloading anything into the site is forbidden. But you still expect Joomla! Update to work. But it downloads code AND installs it to make it executable. Cognitive dissonance much?

But anyway, URL fopen without URL include means that you cannot include or require a malicious file over a URL which was the original reason people were disabling it. So here is your careful whitelist: it has zero entries at all times. Still, downloading stuff and running stuff are different things.

Once more, read my darned article, I am merely echoing what the people developing PHP are recommending to people like you. Surely they know much better?!!!

But no, you do not trust these people. Instead you want to invest a significant amount of time to create a memory hog which will fail out cold on half of the grossly misconfigured servers which have disabled BOTH URL fopen wrappers AND cURL. Something which according to your idea of server security must not exist because it allows code from an arbitrary URL to be downloaded and installed in a way that makes it executable WITHOUT whitelisting and WITHOUT anti-tampering protections. Insane.

What you need to do in your paranoia is NOT use Joomla! Update instead of trying your best to break it for everyone else.

Also, if you are going to be anal retentive about something please be consistent and do not screw over other people's sites in the process. And please don't imply that I didn't know what I was doing when I wrote Joomla! Update. Everyone can see that there is only one moron who doesn't get PHP in this conversation and that isn't me, pal.

Now can we close this idiotic non-issue and move on?

I have to reply from a mobile phone so excuse me for not at mentioning you, OP.

No problem.

If you can still not understand what is the difference between open and include please RTFA until you do. Do you seriously disagree with the people who wrote PHP about open vs include?

Yes, I do, as any other sane person does, possibly excluding you, but certainly not the people who made this mess.

I mean, sure, ignore me and my article. Just read the PHP documentation. Where do you think my information comes from? I don't talk to God, I just RTFM.

Are you sure you don't?

The documentation is misleading on the verge of malicious disinformation. Perhaps whoever wrote it was an NSA plant? Perhaps you are?

To spare you some trouble. URL fopen wrappers, cURL and raw sockets can all be used to download data, no matter what it is. Sure you can disable them all, but you are just being a twat because I bet you do have something that supports uploads, eg media manager, which can be theoretically exploited and don't get me started on your server's security.

Ever heard of "reducing the attack surface"?

According to your idea of security, downloading anything into the site is forbidden. But you still expect Joomla! Update to work. But it downloads code AND installs it to make it executable.

Please point out to me where in my original bug report I mentioned a single word about expecting it to work. I merely wrote that I found it lying to my face and, perhaps ill-advisedly, left it to the reader to conclude that I wanted it to stop doing that.

But anyway, URL fopen without URL include means that you cannot include or require a malicious file over a URL which was the original reason people were disabling it. So here is your careful whitelist: it has zero entries at all times. Still, downloading stuff and running stuff are different things.

Yes, they are. Let me try to explain it to you again:

• allow_url_include enables downloading stuff and running stuff in one step. Bad.
• allow_url_fopen enables downloading stuff. Running it needs a second step. Very nearly as bad.

Once more, read my darned article, I am merely echoing what the people developing PHP are recommending to people like you. Surely they know much better?!!!

The people developing PHP? Know about *dissolves in giggles* security? Ooh, that's a good one.

But no, you do not trust these people. Instead you want to invest a significant amount of time to create a memory hog which will fail out cold on half of the grossly misconfigured servers which have disabled BOTH URL fopen wrappers AND cURL.

Pardon me, "memory hog"? Are you saying your code is so buggy that it turns into one when run on a system where its unvalidated assumptions about the execution environment are not fulfilled?

What you need to do in your paranoia is NOT use Joomla! Update instead of trying your best to break it for everyone else.

I did no such thing, as you could have easily found out if you had spent even a single minute considering my request rather than jumping to the defense of your -- plainly indefensible -- code.

The manual page which per this idiot is written my NSA and gives malicious disinformation is http://php.net/manual/en/filesystem.configuration.php http://php.net/manual/en/filesystem.configuration.php

Let me paste the "malicious disinformation" written by "NSA plants":

allow_url_fopen boolean http://php.net/manual/en/language.types.boolean.php
This option enables the URL-aware fopen wrappers that enable accessing URL object like files. Default wrappers are provided for the access of remote files http://php.net/manual/en/features.remote-files.php using the ftp or http protocol, some extensions likezlib http://php.net/manual/en/ref.zlib.php may register additional wrappers.

allow_url_include boolean http://php.net/manual/en/language.types.boolean.php
This option allows the use of URL-aware fopen wrappers with the following functions: include http://php.net/manual/en/function.include.php,include_once http://php.net/manual/en/function.include-once.php, require http://php.net/manual/en/function.require.php, require_once http://php.net/manual/en/function.require-once.php.

No further comments.

Yes. You will notice that not even in the description of ...include does it waste a single word on mentioning security considerations. So whence comes your confidence that ...fopen has none?

Also, whoever finds hyperbole in my previous post is welcome to keep it.

@chrullrich the allow_url_fopen is unrelated to the fact your picking up the wrong latest version. That comes from a completely separate part of the Code to @nikosdion 's update code (specifically JUpdater). However you might have issues with the result being cached. If you click the purge cache button in Joomla Update does that fix the issue?

Read the fine manual http://php.net/manual/en/function.include.php Pasting it:

The include statement includes and evaluates the specified file.

The documentation below also applies to require.

Files are included based on the file path given or, if none is given, the include_path specified. If the file isn't found in the include_path, include will finally check in the calling script's own directory and the current working directory before failing. The include construct will emit a warning if it cannot find a file; this is different behavior from require, which will emit a fatal error.

If a path is defined — whether absolute (starting with a drive letter or \ on Windows, or / on Unix/Linux systems) or relative to the current directory (starting with . or ..) — the include_path will be ignored altogether. For example, if a filename begins with ../, the parser will look in the parent directory to find the requested file.

For more information on how PHP handles including files and the include path, see the documentation for include_path.

When a file is included, the code it contains inherits the variable scope of the line on which the include occurs. Any variables available at that line in the calling file will be available within the called file, from that point forward. However, all functions and classes defined in the included file have the global scope.

So, the fine manual tells you that include and require will load up a bunch of code found in the location you specified and run it as though it was part of the file where the include statement is included. If you cannot understand the security implication of including and running code I give up, as you should do with coding. Go educate yourself.

@wilsonge No, it doesn't. But please note that my intention in configuring the server was to ensure it could not retrieve remote resources, which clearly must prevent it from detecting any updates at all. In this situation, I would have expected a message like "update check failed, please look at joomla.org to see if a newer version exists". Instead, I get "you are running the latest version", even including the (non-latest) installed version number in the message.

OK so what happens if you go into Extension manager update view and click on "Purge" there (it should do the same thing but just humour me)

Lets ignore the paranoid web settings and the unwarranted personal attacks.

My understand is that @chullrich wants a message such as "sorry I dont have
a clue what the latest version is as I cant see anything outside of this
server" on the updates

If that is correct then this would also be required in the language
manager, install from web, extension update, numerous 3pd extensions and
perhaps somewhere else I cant think of.

As I said earlier if you want to submit code to provide that message then I
am sure it will be considered. However I wouldnt expect anyone to write
that code for you. You have chosen a level of paranoia server settings that
significantly reduce the functionality of Joomla so if you want Joomla to
still work then it is your responsbilty to provide the code. Attacking the
volunteers that have contributed hundreds and thousands of hours to provide
you with Joomla is not the way to get someone to do something for you.

On 10 July 2015 at 08:25, chrullrich notifications@github.com wrote:

@wilsonge https://github.com/wilsonge No, it doesn't. But please note
that my intention in configuring the server was to ensure it could not
retrieve remote resources, which clearly must prevent it from detecting any
updates at all. In this situation, I would have expected a message like
"update check failed, please look at joomla.org to see if a newer version
exists". Instead, I get "you are running the latest version", even
including the (non-latest) installed version number in the message.

—
Reply to this email directly or view it on GitHub
#7369 (comment).

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

• Starting fresh, new database, newly unpacked 3.4.2, allow_url_fopen initially off: No update indicated in control panel or Joomla Update.
• After enabling allow_url_fopen: Update to 3.4.3 correctly indicated in both locations, one record in the _updates table for Joomla 3.4.3.
• After disabling it again: No update indicated in control panel or Joomla Update, _updates table is empty.

That last effect, where the update indication disappears again, is something I had not encountered the last time I tested this.

• After clicking "Purge cache" in Joomla Update, with the option still disabled: No update indicated, but there is an error message, "Update: Could not open update site #1 "Joomla! Core", URL: http://update.joomla.org/core/list.xml"

This is something at least, although it may not be obvious to everyone that it means the no-update indication below is wrong. After all, it may have reached some "update site #2".

@brianteeman I think the application should not strive to create a false sense of security in its user, if anything, it should do the opposite. My configuration may "significantly reduce the functionality of Joomla", yes, although it is a matter of opinion. @nikosdion [Edit: Apologies, @brianteeman,] pointed out to me that the requirement is documented, so it is not entirely unreasonable to expect the user to be aware of it.

Again, I do not complain that it does not work, only that it doesn't report that it does not work.

Joomla has surely had its share of vulnerabilities, enough that you (volunteer developers as a whole) put in the work to add an update detection and an update installation feature. Do you really believe that this is of so low a priority that having it silently fail is not a cause for concern? As I just told @wilsonge, even the error message about an unreachable update site only appears after purging JU's cache, not every time that page is opened. So unless the user is already suspicious about the possible availability of an update, and forces JU to recheck, they will never be told anything but "all is well".

And, finally, I don't think disabling URL fopen is "paranoid", but then, if I am paranoid, I wouldn't know, would I?

Joomla has surely had its share of vulnerabilities, enough that you (volunteer developers as a whole) put in the work to add an update detection and an update installation feature

No the feature is there for bug fixes. I can count the number of real
vulnerabilities over the last 10 years on my fingers

As I said earlier if you want to submit code to provide that message then I am sure it will be considered. However I wouldnt expect anyone to write that code for you

@brianteeman On the fingers of three hands exactly, if you only count the "critical" and "high" advisories, and the one marked "medium high". Including the lower-ranked ones I think there are 71 right now.

Some facts.

Out of the 27,309 unique sites current active and connected to myJoomla.com, 27,227 of them allow file_get_contents to grab a URL.

Only 82 out of the 27,309 sites do not allow file_get_contents to grab a URL and so have to use curl

Zero sites are left that are forced to use raw sockets/streams or other hacks to access remote urls.

So lets keep this in context ...

So, I'd recap what we've learned so far:

• it's possible that in restricted config circumstances (URL wrappers turned off) the messages about available updates give misleading information; someone has to write code for it
• allow_url_fopen = 1 is a hot topic, and it will remain as such for sure. But the "legendary" fact that Joomla requires allow_url_fopen = 1 isn't true. So, turning it off is really a security preference choice and that was my point. I gave my reasons why I think it should be off, and Joomla will still work for me. And I'd stop at that, everyone is free to choose their comfortable security/"paranoia" level.

Seeing how this topic degenerated into personal attacks and "names calling" for OP and me (Twitter and here), I am sorry that I tackled the related problem of URL wrappers and Updater component missing raw socket way of downloading it in this topic. Still, I am sure that would be a good additional option which wouldn't harm anyone, but would still help in "tight security" server setups. I'll prepare a separate pull request with a separate ticket, and then it's up to others to decide if it works.

Gentlemen, this has been a wonderful read regarding sever / site security, relating to the joomla updater so thank you ALL for this knowledge.

Back to the point of this topic and helping others who may have same / similar problems (thank you for taking on the topic btoplak).
I recently migrated from an old server, previous sys admin to a new server, new sysadmin who is also extremely concerned with security to the point of paranoia (which is why we use him, we are too) and I have spent countless hours tracking down both the cause and solution to related issues. My two cents is that messaging like the following (even if was meant to be sarcastic)

"sorry I dont have a clue what the latest version is as I cant see anything outside of this server"

would actually be extremely beneficial for noobs and non sysadmin types like myself who end up in a security-rich hosting environment. One of our sites is a JomSocial based community (closed) for the medical industry so the last thing we want to do is to increase our attack vector. I'll admit it... I am paranoid of being hacked or compromised! ..but I still need Joomla, K2, Kunena, AlphaUserPoints, Akeeba, JoomlaUpdater, RSFirewall, and JomSocial to function. I'd love even more to know the failures I seem to be having are due to heightened hosting security and server hardening. So again, thank you to all for the info, it has solved a problem that has consumed many of my waking hours!

It has been over a year since this topic was raised. No one has proposed any code to address the alleged issue. Sorry but I am not in favour of just leaving things open for ever so I am closing it due to lack of interest. If anyone wants to submit a PR then it will of course be considered but just keeping this open any longer serves no purpose