I have tested this item successfully on ce5fb57

Test was successful on mysql

I have tested this item successfully on ce5fb57

Install local, the Joomla! updates sites use https like described :)

The com_joomlaupdate and files_joomla extension manifests need to be updated as well. There is also some code in components/com_users/controllers/profile.json.php calling the update server to fetch the helpsite list that need fixing.

Finally there are also some calls in FOF 2 - @nikosdion how are we dealing with that as FOF2 is now no longer actively maintained? do we just do it in core or do you still want it in the fof 2 branch of the fof repo?

This PR has received new commits.

CC: @n9iels, @peterpeter

Thanks @wilsonge done. @nikosdion let me know what should updated and I'm happy to do. If nothing should be updated or you will send your own PR please let us know. Thanks ;)

Due to the new commits, the test-instructions(First Test) stays the same? Or is there something more to check?

Just an EXTREMELY IMPORTANT heads up!!! Updates are already broken for many servers. Since you're adding HTTPS support do it right ;)

Currently you are using the cacert.pem distributed by haxx.se (the makers of cURL), i.e. curl.haxx.se/ca/cacert.pem THIS WILL NOT WORK WITH JOOMLA! UPDATE ON ALL SERVERS

The cacert.pem distributed by haxx.se is derived from Firefox' source code. Since early 2015 Firefox has dropped certain CAs which only have an SHA-1 signature. Unfortunately, one of those is a Verisign root CA which happens to be the signer of Amazon S3's SSL certificates. GitHub uses Amazon S3 to store and server downloads. Joomla! uses GitHub to distribute downloads. Therefore Joomla! indirectly uses Amazon S3.

Since the root CA for the HTTPS download URL is missing from cacert.pem, if you are on a server which does not have this root CA in the system wide path visible to cURL the peer validation fails and the update cannot continue. There are two solutions:

1. The BAD way (do NOT implement!): turning off peer validation. This is tantamount to using plain HTTP and allowing MITM attacks. It's utterly stupid and the only reason I'm typing it is to stress out how moronic it would be implementing it, pre-empting anyone who might complain "but the easy way is to turn off peer validation".

2. Use a root CA cache which includes the root CAs used by Amazon S3. I have compiled a cacert.pem from CentOS' root CA cache. Here's a copy for your convenience: https://github.com/akeeba/fof/blob/development/fof/Download/Adapter/cacert.pem

As for testing that... you need to find a broken server :p It took me a few months to find a client with an affected server to test my theory on what's going wrong and the fix.

Also see http://issues.joomla.org/tracker/joomla-cms/8193

@brianteeman This is exactly what I said. Let me repeat myself:

The cacert.pem distributed by haxx.se is derived from Firefox' source code. Since early 2015 Firefox has dropped certain CAs which only have an SHA-1 signature. Unfortunately, one of those is a Verisign root CA which happens to be the signer of Amazon S3's SSL certificates. GitHub uses Amazon S3 to store and server downloads. Joomla! uses GitHub to distribute downloads. Therefore Joomla! indirectly uses Amazon S3.

Since the root CA for the HTTPS download URL is missing from cacert.pem, if you are on a server which does not have this root CA in the system wide path visible to cURL the peer validation fails and the update cannot continue.

I didn't say it's a new issue. I'm saying that now that you're out of JoomlaCode.org (a plain HTTP site) and into an HTTPS Amazon S3 bucket you need to make sure your clients won't have a problem updating. I think it's important that everyone be able to update Joomla! with one click and no trouble doing that. Right?

My comment was for others not you @nikosdion
On 2 Nov 2015 11:34 am, "Nicholas K. Dionysopoulos" <
notifications@github.com> wrote:

@brianteeman https://github.com/brianteeman This is exactly what I
said. Let me repeat myself:

The cacert.pem distributed by haxx.se is derived from Firefox' source
code. Since early 2015 Firefox has dropped certain CAs which only have an
SHA-1 signature. Unfortunately, one of those is a Verisign root CA which
happens to be the signer of Amazon S3's SSL certificates. GitHub uses
Amazon S3 to store and server downloads. Joomla! uses GitHub to distribute
downloads. Therefore Joomla! indirectly uses Amazon S3.

Since the root CA for the HTTPS download URL is missing from cacert.pem,
if you are on a server which does not have this root CA in the system wide
path visible to cURL the peer validation fails and the update cannot
continue.

I didn't say it's a new issue. I'm saying that now that you're out of
JoomlaCode.org (a plain HTTP site) and into an HTTPS Amazon S3 bucket you
need to make sure your clients won't have a problem updating. I think it's
important that everyone be able to update Joomla! with one click and no
trouble doing that. Right?

—
Reply to this email directly or view it on GitHub
#8196 (comment).

The Joomla Updater doesn't check the full chain of certificates.
See https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_joomlaupdate/helpers/download.php#L184
and https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_joomlaupdate/helpers/download.php#L226

So this should not be an issue at all.

By the way, we already ship a cacert.pem in the current version of Joomla. It's the cause why extension updates fail if the file is hosted on GitHub (eg weblinks). By default our curl requests are checking the full chain (which is actually the right thing to do).

I'm not sure what AmazonS3 did, but the cert seems to be valid now till the end of year. Earlier this year when I check it was to be expire by end of october or so. See https://www.sslshopper.com/ssl-checker.html#hostname=github-cloud.s3.amazonaws.com

Yes Nic is talking about the cacert.pem that we ship with in the entire of his message above!!! I think you need to reread it @Bakual :P

I even offered a replacement cacert.pem which solves the issue, Thomas :)

@wilsonge Yes, but we already have peer validation turned off for com_joomlaupdate (for a long time!). It's the option 1 Nicholas said to not do (where I agree) but we do that already since 2.5.x.
See 4dca060#diff-fe8174b006820db6f9ad1eba5ec202b2R182 and 4dca060#diff-fe8174b006820db6f9ad1eba5ec202b2R222.

Apparently we had that code because we never bothered with HTTPS. Of course we have to fix that BUT also fix the cacert.pem as well.

So what we need to do is convert that code to use JHttp AND also fix our shipped version of cacert.pem

All I say is that Joomla Updater will continue to work.
A custom cacert.pem will of course fix the extension updates. But then, I still think the real fix is that Amazon should fix their certs. So I would see the custom cacert.pem as a workaround until Amazon fixes their stuff.

the Joomla Updater will NOT work after this patch if we use HTTPS because as you point out we are not loading any certificates. And the existing certificates don't cover us

For Christs sake it's not custom! It's what CentOS ships with the distro that's been used by half of the hosts out there... Also, what George already said. If you don't replace the file you are heading for yet another massive Joomla! Screw up. I am trying hard to let the project save face.

I'm not saying not to do that.
I don't know the exact reasoning why we have turned off the peer validation in the current code (maybe @nikosdion just copied it from the existing code back then). It's certainly not a secure way but probably worked better for stupid hosts.

As for "custom" or "fixed", doesn't matter to me. All I wanted to say is that it's not the original one (from the cURL makers) we usually use. If we go that route, we need to make sure that we don't revert it in a future update by accident.

Just on a sidenote: This PR doesn't have anything to do with the GitHub/AmazonS3 cert issue. It activates https for a server of our own (update.joomla.org) which serves the update stream XML (eg https://update.joomla.org/core/list.xml).
The update packages themself are delivered from GitHub/AmazonS3) using https for quite some time already.

I know Thomas. I have seen the people who have issues updating and was trying to track down why. It took a client with an affected server to let me do that. Once I did it for my software it made sense why Joomla update also fails. By switching the update stream to HTTPS you may end up in the same situation depending on who is signing your cert. I told you how and why and how to fix it. If you wanna ship broken software I can not stop you but at least my clear warning is on public record. Good night.

Moving to Needs Review so the PLT / Maintainer have the last call to close or not.

This doesn't need to be closed. But it does require the fixes that Nic has mentioned before it can be merged

Personaly i think this is a different issue the CMS update server is not hosted on github we have our own cert there and the CMS Updater has disabled the cert checks. But if someone wants to take that over: Here is the code: https://github.com/zero-24/joomla-cms/tree/updateServerHTTPS feel free to extend and send a clean PR that fixes the issue. And updates if needed also the fof files.

@wilsonge This really is unrelated to the issue Nic mentioned.

Folks you realize one of the reasons this project is a joke outside your
circles is because of things like this. There is an Internet wide push for
HTTPS communications and if you can't manage to put egos or finger pointing
aside to secure your update channels (something folks have pointed out is a
security issue of sorts for some time), you might as well just rip out the
updater.

The Amazon certificate issue does not go away by updating their
certificates. Whether packages are hosted on GitHub or on your stalled out
downloads site, Amazon is the CDN service. So you MUST make it work with
Amazon or host your own CDN service. And yes, the issue of pulling the
package from there is disconnected from just getting Joomla to ping
update.joomla.org on a SSL connection.

So when you're done trying to have a power play or shrug off legitimate
issues by pointing fingers at third parties, get this branch updated and
merged. Sooner than later.

On Friday, November 6, 2015, Thomas Hunziker notifications@github.com
wrote:

@wilsonge https://github.com/wilsonge This really is unrelated to the
issue Nic mentioned.

—
Reply to this email directly or view it on GitHub
#8196 (comment).

Reopening since this is still relevant.

I've already told you how to fix it (hint: https://github.com/akeeba/fof/blob/development/fof/Download/Adapter/cacert.pem) It literally takes all of 2 minutes to do the commit. For crying out loud, fix your goddamn cacert.pem file! It will also MAGICALLY fix 3PD extensions updates that mysteriously fail on some servers when the 3PD is using HTTPS. I have such a component in mind on my personal site.I have to remember to replace cacert.pem every time I update Joomla! – which is annoying. Just commit the fix already!

@nikosdion is there a URL from CentOS where we can download updates to this cert from? (for maintenance in the future - obviously for this PR we can use the one in FOF)

See
https://techjourney.net/update-add-ca-certificates-bundle-in-redhat-centos/

On 18 November 2015 at 01:09, George Wilson notifications@github.com
wrote:

@nikosdion https://github.com/nikosdion is there a URL from CentOS
where we can download updates to this cert from? (for maintenance in the
future - obviously for this PR we can use the one in FOF)

—
Reply to this email directly or view it on GitHub
#8196 (comment).

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

@wilsonge its also documented on the chef repo how to do it but it looks like you have to convert the certificate bundle with a flag that the pre-created bundle doesnt use itself.

@nikosdion @wilsonge I already told you several times that this PR here is absolutely unrelated to the cacert.pem (GitHub/Amazon) issue.

Please open a new PR or comment on the one who wants to update the cacert.pem. It's appropriate there and I actually agree on the proposed fix. But it's unrelated to this one here.

No, it is entirely related! You can't enable HTTPS updates without making sure they can be downloaded and installed by Joomla! Update. If you merge this PR without fixing cacert.pem you are making sure that a significant percentage of users will NOT be able to install security updates. Since you do NOT have a safeguard in place to only merge this PR if another PR isn't merged you MUST include both fixes in the same PR. Otherwise you will inevitably screw up the updates. It's called common sense and I've grown tired of debating it. Please don't at-reply to me. I'm unsubscribing from this thread. Do whatever you want.

Sorry @bakual you are wrong. I did a quick google last night to try and
answer the question of @wilsonge last night and there are LOTS of
references to this problem with https and anything stored on AWS when you
are using the cacert.pem provided by firefox. It is even discussed and
confirmed on the site that we are getting the cacert from. We can NOT
switch to https without changing the cacert to one that does not exclude
sha1

On 18 November 2015 at 08:00, Nicholas K. Dionysopoulos <
notifications@github.com> wrote:

No, it is entirely related! You can't enable HTTPS updates without
making sure they can be downloaded and installed by Joomla! Update. If
you merge this PR without fixing cacert.pem you are making sure that a
significant percentage of users will NOT be able to install security
updates. Since you do NOT have a safeguard in place to only merge this PR
if another PR isn't merged you MUST include both fixes in the same PR.
Otherwise you will inevitably screw up the updates. It's called common
sense and I've grown tired of debating it. Please don't at-reply to me. I'm
unsubscribing from this thread. Do whatever you want.

—
Reply to this email directly or view it on GitHub
#8196 (comment).

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

@brianteeman Yeah, but the update streams which is changed here AREN'T stored on Amazon. They are on our OWN servers (update.joomla.org) with our OWN cert.

This PR is not talking about the update PACKAGES which are hosted on GitHub/Amazon (and already are served using HTTPS). I never said the issue raised by @nikosdion is invalid. I said it's unrelated to this PR.

Unsubscribed
On 18 Nov 2015 9:26 am, "Thomas Hunziker" notifications@github.com wrote:

@brianteeman https://github.com/brianteeman Yeah, but the update
streams which is changed here AREN'T stored on Amazon. They are on our OWN
servers (update.joomla.org) with our OWN cert.

This PR is not talking about the update PACKAGES which are hosted on
GitHub/Amazon (and already are served using HTTPS). I never said the issue
raised by @nikosdion https://github.com/nikosdion is invalid. I said
it's unrelated to this PR.

—
Reply to this email directly or view it on GitHub
#8196 (comment).

sigh...

The actual update packages have been downloaded from an HTTPS URL for about a year now, this PR does not change that one bit. Users reaching SSL errors if updating outside the update component before this PR would continue getting them after this PR without any additional action. JoomlaCode quite simply stopped being able to handle the amount of traffic that was thrown at it and since the middle of the 3.3 series GitHub (and by proxy Amazon S3) have been the primary host for Joomla downloads and the location the packages are pulled from when updating within Joomla.

What this PR does change is the URL that JUpdater queries via JHttp to retrieve the relevant XML file. If the SSL certificate that update.joomla.org has assigned to it does not validate with the current cacert.pem file, then that needs to be addressed.

There are two totally separate issues in this thread. Both of them 100% valid, neither is dependent on the other to be merged. Both items need to be acted on, but the act of querying for the update XML stream and the download of a ZIP file are two independent events using two totally separate code paths if you are using the update component.