User tests: Successful: Unsuccessful:
This is an update for pull request #7677
Reverse caching proxies like Varnish, Nginx etc. (also known as web accelerators) are built to cache the entire page output of a website and thus speed up delivery significantly. Such software works much more efficiently to how page caching is performed internally in Joomla, as PHP & the database (the usual performance bottleneck) are invoked only when the content is required to be constructed and not every time it's served.
The use of such software is not only meant to be used by extremely high performance sites, as even sites with low or moderate traffic can benefit from the speed improvement and thus rank better in search engine results.
Joomla unfortunately cannot work out-of-the-box with reverse caching proxies. This limits Joomla from being easily setup in high performance scenarios without additional custom coding or extensions, unlike other CMSs like WordPress, which already provide the means for reverse caching proxies to operate better with.
There are currently 3 problems that prevent Joomla from working out-of-the-box with reverse caching proxies:
a) sessions for logged in users cannot be easily identified by that software, in other words, a reverse caching proxy is unable to "know" if a user is logged into Joomla and therefore cannot decide whether to serve cached or un-cached output. This obviously has undesired effects where guest users could see sensitive data cached from a previously logged in user.
b) the "remember me" cookie cannot be identified as well (because it's a hashed user agent string) and thus the related functionality is simply broken, even if we fixed (a).
c) Identifying which pages NOT to cache, even when the above conditions are never met. Such a page is the typical user login page and the problem comes from the token used in the login form. Because the token is created from a guest user's session cookie, if the login page (or any page that requires user login - e.g. a user module, a members area etc.) has been previously cached by the reverse caching proxy, then there will be a token mismatch between what gets submitted to the form and the token the actual user would have generated by Joomla if there was no caching implemented at all. That's because reverse caching proxies cache some page from the first visitor that accesses that page. So when a 2nd visitor comes in, the page would essentially display what Joomla generated for the first visitor. And in the case of forms, the 2nd visitor would get the 1st visitor's session token printed on the html of that form. A token mismatch means the visitor could not login without re-submitting the form, which translates to unnecessary user experience issues.
The solution is overall simple and it requires the use of some identifiers in the form of cookies or HTTP headers:
For (a), all we have to do is set an identifier cookie only when users log in. This cookie only acts like a flag telling reverse caching proxies that as long as this cookie is present, the output of the page should never be cached. No sensitive data is exposed and this cookie is destroyed when the user logs out.
For (b), all we have to do is add a prefix to the existing cookie created by Joomla, which is currently a hashed user agent string. This allows reverse caching proxies to check for this prefix and turn off any caching, similar to how it should be done for when users log into a Joomla site.
For (c), the users component (com_users) and the entire Joomla /administrator path should emit an HTTP header which would instruct the reverse caching proxy to NOT cache content at all and therefore allow the Joomla form token to be created from the actual guest attempting to log in. These are the 2 most important user entry points in Joomla. Unfortunately, we cannot count the user login module to this change, because if that module is used on every page of the site, then the HTTP header to prevent caching would emit everywhere and thus beat the point of caching overall. It's a case by design which must be addressed by the developer of each site. As an example, see how we handle user logins at http://www.joomlaworks.net which uses Varnish. You'll notice that every user login leads to a specific page only. This page is excluded from being cached by Varnish and therefore user logins work flawlessly.
The proposed PR currently fixes (a) and (b) which is the most important aspect of Joomla's out-of-the-box interoperability with reverse caching proxies. Issue (c) is less important as it can easily be addressed using an existing boilerplate configuration for Varnish, Nginx etc. When I have a user-configurable solution for this, e.g. a flag in the Joomla menu system to exclude pages from such caching by emitting a special HTTP header, I will make a new PR with required code changes.
To sum things up:
Thank you.
P.S. This is the configuration for Varnish v3.x to detect the "user state" and "remember me" cookies in Joomla properly (it's really typical cookie detection in Varnish):
sub vcl_recv {
[...other rules...]
if (req.http.Cookie ~ "joomla_") {
return (pass);
}
[...other rules...]
}
and
sub vcl_fetch {
[...other rules...]
if (bereq.http.Cookie ~ "joomla_") {
return (hit_for_pass);
}
[...other rules...]
}
A full example can be found here: https://gist.github.com/fevangelou/84d2ce05896cab5f730a
P.S. The use of "joomla__" (instead of "j__" as naming conventions are usually used in Joomla) was done for clarity. WordPress for example uses "wp_" or "wordpress_" and other CMSs similar more-than-one-letter naming conventions. If we used just "j_*" that might have undesired effects as the cookie name match may be more common.
To see the full benefits, use Joomla installed on a LAMP stack with Varnish 3.x and this configuration as a starting point: https://gist.github.com/fevangelou/84d2ce05896cab5f730a
Otherwise just get the update, open up your browser's dev tools and inspect the cookies set when you log in and out and when you tick the "remember me" checkbox in the login forms.
/cc @Hackwar @brianteeman
Status | New | ⇒ | Pending |
Labels |
Added:
?
|
Category | ⇒ | Cache |
Rel_Number | 0 | ⇒ | 7677 |
Relation Type | ⇒ | Pull Request for |
That's security through obscurity which is no security at all
There is also the language cookie in multilangue sites.
https://github.com/joomla/joomla-cms/blob/staging/plugins/system/languagefilter/languagefilter.php#L756
That's security through obscurity which is no security at all
Actually is to try to avoid robots trying to check the sites.
There are a million other ways a robot can identify a joomla site
I know that. But we can avoid some of the must common. That is just my opinion. Anyone is free to disagree.
Nevertheless, just want to say that this PR is a good improvement.
Actually if you look at the hacker scripts this is rarely used as there are
easier methods that will also reveal the version number which is far more
useful.
@brianteeman as i said, is my opinion. We can agree to disagree on this.
@fevangelou can you do the same thing for the language cookie?
https://github.com/joomla/joomla-cms/blob/staging/plugins/system/languagefilter/languagefilter.php#L756
It's not required to have language specific cookies as Joomla uses
different URLs either way and thus there is no issue with multilingual
sites.
BTW in my description I clearly explain why there is no security issue by
using the joomla_ prefix. Furthermore it should not be user configurable as
this way it would beat the whole purpose of it in the first place.
@brianteeman is everything OK to be merged? I can then work on a solution
for problem (c).
On May 10, 2016 1:46 PM, "andrepereiradasilva" notifications@github.com
wrote:
@brianteeman https://github.com/brianteeman as i said, is my opinion.
We can agree to disagree on this. [image: ]@fevangelou https://github.com/fevangelou can you do the same thing for
the language cookie?—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#10373 (comment)
i read it now. You're only talking about the remember me cookie and a new cookie with the user login state.
I tough this PR was for the session cookie. Sorry i misunderstood.
I have tested this item successfully on 856b264
I have tested this item successfully on 856b264
Status | Pending | ⇒ | Ready to Commit |
Thank you @fevangelou, can you also provide the solution for the admin part as well? I am guessing some code in the JApplicationAdministrator in another PR
Labels |
Added:
?
|
@brianteeman can we get a milestone here? ;)
Milestone |
Added: |
Labels |
Added:
?
|
I have just send a quick CS PR. fevangelou#1
This PR has received new commits.
CC: @andrepereiradasilva, @dgt41
This PR has received new commits.
CC: @andrepereiradasilva, @dgt41
Status | Ready to Commit | ⇒ | Fixed in Code Base |
Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2016-05-10 19:36:32 |
Closed_By | ⇒ | rdeutz |
Labels |
Removed:
?
|
This will be merged into the code of Joomla 3.6?
If is not sure, but it is only a matter of help test production or make adjustments, I am committed to help with this. Please, just let me know.
cc @bcdonadio
It's already merged :)
Regards,
Fotis Evangelou
On 1/7/16 1:07 πμ, Emerson Rocha Luiz wrote:
This will be merged into the code of Joomla 3.6?
If is not sure, but it is only a matter of help test production or
make adjustments, I am committed to help with this. Please, just let
me know.cc @bcdonadio https://github.com/bcdonadio
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10373 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABPdGzabJDAXL6gJ25oO-zwcNNT_G_1sks5qRD4vgaJpZM4IatKH.
@fititnt you Can test with The RC1 of 3.6 https://github.com/joomla/joomla-cms/releases/tag/3.6.0-rc :)
Hi all,
To make the most of this PR, do we need to:
Thank you for your feedback.
@fred-the-coder everything you need is described in detail here https://gist.github.com/fevangelou/84d2ce05896cab5f730a (updated Sept 21st, 2016).
@fevangelou
Hi all, great job, thanks for sharing this...
But there is very important case;
Exclude for module positions and signed-in users cache version, are requirement to complete the cache solution, Joomla and Reverse Cache Proxies (Varnish, Nginx FastCGI) ...
I'm testing for use is same time from Joomla File-Cache (or Memcached [Storage]) and Varnish (or NginxFastCGI).
To support everything together (maybe).
@fevangelou : Where can I find a link for nginx configuration to implement this? Thanks in advance.
Hi all,
I set-up a mock-up for my customer here: http://ra.mosoft.fr
It is a site of news kind where trafic will be high.
Thus, I use Joomla 3.6.x (actually 3.6.2) to benefit from Varnish on my hosting environment.
But, I am afraid Varnish is not working...
I did run this command line:
curl -I http://simple.gandi-test.fr/
HTTP/1.1 200 OK
Server: Apache/2.4.18
X-Powered-By: PHP/5.4.45-0+deb7u2
X-Logged-In: False
X-Content-Powered-By: K2 v2.7.1 (by JoomlaWorks)
Expires: Wed, 17 Aug 2005 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: 8c344a4d64c96915ff93f25331a22a1d=n8i2emphg2ohkqoik9suhslkn3; path=/; HttpOnly
Last-Modified: Mon, 21 Nov 2016 21:15:59 GMT
Content-Type: text/html; charset=utf-8
Vary: Accept-Encoding
Date: Mon, 21 Nov 2016 21:15:59 GMT
Connection: keep-alive
Via: 1.1 varnish
Age: 0
The reported "Age" is "0", meaning cache is not working...am I wrong?
If not, what is wrong here?
Thank you for your help.
Before you do ANYTHING else please update to the latest release which is
3.6.4 - there are serious vulnerabilities in 3.6.2
On 29 November 2016 at 08:23, fred-the-coder notifications@github.com
wrote:
Hi all,
I set-up a mock-up for my customer here: http://ra.mosoft.fr
It is a site of news kind where trafic will be high.
Thus, I use Joomla 3.6.x (actually 3.6.2) to benefit from Varnish on my
hosting environment.But, I am afraid Varnish is not working...
I did run this command line:curl -I http://simple.gandi-test.fr/
HTTP/1.1 200 OK
Server: Apache/2.4.18
X-Powered-By: PHP/5.4.45-0+deb7u2
X-Logged-In: False
X-Content-Powered-By: K2 v2.7.1 (by JoomlaWorks)
Expires: Wed, 17 Aug 2005 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
pre-check=0
Pragma: no-cache
Set-Cookie: 8c344a4d64c96915ff93f25331a22a1d=n8i2emphg2ohkqoik9suhslkn3;
path=/; HttpOnly
Last-Modified: Mon, 21 Nov 2016 21:15:59 GMT
Content-Type: text/html; charset=utf-8
Vary: Accept-Encoding
Date: Mon, 21 Nov 2016 21:15:59 GMT
Connection: keep-alive
Via: 1.1 varnish
Age: 0The reported "Age" is "0", meaning cache is not working...am I wrong?
If not, what is wrong here?Thank you for your help.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10373 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPH8fqDH085eCF-zHF3CIYsdA7ET2nfks5rC-ENgaJpZM4IatKH
.
--
Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
https://brian.teeman.net/ http://brian.teeman.net/
@azeemdwt A good starting point for Nginx, containing the exclusions recently added in Joomla can be found here: https://github.com/engintron/engintron/blob/master/nginx/proxy_params_dynamic
@fred-the-coder Provided you're using the config file I'm suggesting here https://gist.github.com/fevangelou/84d2ce05896cab5f730a (which BTW I updated today with additional improvements for both Varnish 3.x & 4.x), the first hit on your demo site will correctly report Age: 0 if no other hit came before it. For every subsequent one (and for the TTL of the cache) it should provide a value. You can compare with joomlaworks.net (using curl as you did already) and see for yourself.
@fevangelou : Thanks a lot for the link.
I'm llocking here now. Any more questions can be moved to the forum: https://forum.joomla.org or the mailing lists as this is a bug tracker and no support forum ;). Thanks for understanding.:
https://groups.google.com/forum/#!forum/joomla-dev-cms
https://groups.google.com/forum/#!forum/joomla-dev-general
didn't read all the post, but i think i know the problem since i have it to.
But one thing that i don't agree: adding "joomla" text in cookies it will make very easy to detect the server is using joomla.
Can we have a param to costumize that, like
cookie_prefix
(defaulting tojoomla
)?