?
avatar feltkamptv
feltkamptv
23 Dec 2017

On large spikes in user traffic our database gets stuck due to the session handler in Joomla. We get hundreds of the same query waiting for the lock to be released on the rows in MySQL. This is seen with show processlist:

DELETE
FROM vk9wz_session
WHERE time < '1514040959'

We have a few million visitors a day on our Joomla site and have continued upscaling our MySQL server, but this does not seem to fix this issue. After a while the max number of connections is hit and the site breaks.

System information

  • Load balanced servers on AWS
  • MySQL server with 64 GB mem and 16 CPU
  • Cloudflare

We are going to look into using Memcached to store the sessions and hopefully that fixes everything. Using database as session handler for a large Joomla! site seems to be an issue.

avatar feltkamptv feltkamptv - open - 23 Dec 2017
avatar joomla-cms-bot joomla-cms-bot - change - 23 Dec 2017
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - labeled - 23 Dec 2017
avatar feltkamptv
feltkamptv - comment - 23 Dec 2017

We use PHP 7.1 btw. We already use Memcached successfully to save other cache items in our site, but does it work for sessions? Because after switching to Memcached for the sessions, we cannot login anymore into frontend and backend.

avatar franz-wohlkoenig franz-wohlkoenig - change - 23 Dec 2017
Category SQL
avatar alikon
alikon - comment - 24 Dec 2017

try using redis for session

avatar franz-wohlkoenig franz-wohlkoenig - change - 24 Dec 2017
Status New Discussion
avatar feltkamptv
feltkamptv - comment - 24 Dec 2017

We will try out redis later. We have now memcached working, unfortunately the sessions table is still being used, but the hundreds/thousands of delete session queries have disappeared. We are still looking into disabling guest sessions without hacking Joomla, which should also help a lot for performance.

Conclusion seems to be for now: do not use database as session handler for sites with lots of traffic.

Happy holidays everyone.

avatar mbabker
mbabker - comment - 24 Dec 2017

#19165 should be able to assist with this.

However, note that Joomla internally is reliant on sessions and therefore it is essentially impossible to disable them in full. Likewise, Joomla arbitrarily logs certain metadata about the session regardless of the handler in use (there is a proposal for 4.0 to be able to turn this off however it hasn't seen any activity in quite some time) therefore there needs to be some logic to clean this data out of the database as it expires too. The linked PR tries to help with that part of things.

avatar feltkamptv
feltkamptv - comment - 25 Dec 2017

Unfortunately Memcached is giving problems too. Users are being logged out/in randomly. We only use a single Memcached instance with ElastiCache because a cluster doesn't work (missing autodiscovery).

We are now writing our own session handler, based on DB, used only for public site and disabled for our webservices.

Is it just as simple as copying the DB session handler file, renaming the file and class and adding it to config?
public $session_handler = 'databasecustom';

How does Joomla.org solve these session issues?

avatar joomdonation
joomdonation - comment - 25 Dec 2017

If I understand your issue correctly, you can try this solution:

  1. Apply the change from PR ##19165 which @mbabker proposed

  2. Change php.ini setting session.gc_probability to 0

  3. Setup cron job to run this PHP function http://php.net/manual/en/function.session-gc.php

Basically, with that setup, the DELETE command which is using to delete session data won't be run base on user's requests (which is a lots because you have high traffic website but base on pre-defined amount of times you configure for cron job. In theory, it will reduce number of DELETE command and hopefully, it will solve your issue.

avatar feltkamptv
feltkamptv - comment - 25 Dec 2017

@joomdonation Thank you for your answer! Even on Christmas! This looks like a very good solution and we are looking into it right now!

avatar joomdonation
joomdonation - comment - 25 Dec 2017

One thing I should mention is that PR #19165 only support database handler now, so you need to back to use database handler if you want to use this solution.

avatar mbabker
mbabker - comment - 25 Dec 2017

How does Joomla.org solve these session issues?

I don't know what the JED or forums do for sessions, but the sites that I'm running are all using the PHP filesystem handler and Joomla's conservative caching enabled, they are handling things pretty well in general.

avatar feltkamptv
feltkamptv - comment - 25 Dec 2017

Most of our Joomla sites are on shared hosting and do also use the filesystem handler, but for loadbalanced sites we use the database and that has been going well, except for this site only which has lots of traffic.

We are now implementing the PR and testing the solution of @joomdonation. The application runs on 130+ servers within 12 server groups, so it's going to take some time to test and deploy.

avatar feltkamptv
feltkamptv - comment - 26 Dec 2017

@joomdonation @mbabker Thank you very much! We deployed the solution to our platform and have a cron job running every minute calling the session GC function. Our database connections are down to low levels again and the site is on fire! We will keep on monitoring everything, but hopefully we do not encounter any issues anymore with this.

Thanks for the quick support (even with Christmas), it is very much appreciated! Happy holidays!

avatar franz-wohlkoenig
franz-wohlkoenig - comment - 26 Dec 2017

@feltkamptv can you please close the Issue as its solved?

avatar feltkamptv feltkamptv - change - 26 Dec 2017
Status Discussion Closed
Closed_Date 0000-00-00 00:00:00 2017-12-26 16:51:53
Closed_By feltkamptv
avatar feltkamptv feltkamptv - close - 26 Dec 2017
avatar joomdonation
joomdonation - comment - 27 Dec 2017

@feltkamptv Good to hear that it helps solving your issue and happy to hear that Joomla can handle that kind of big site.

Just want add that you might want to check your cron job and make sure it clean up expired session data. As I understand (and quick tested), if your cron job just run PHP script like below, it won't trigger gc method of custom session handle which we are using in Joomla (database in this case)

session_start();
session_gc();

To make sure it works, I think your cron job will need to trigger a CLI application in Joomla. I my test, I modified cli/garbagecron.php to the below code, run it and gc method of database storage handler is triggered properly:

public function doExecute()
{
	$session = JFactory::getSession();
	$session->start();
	session_gc();
}
avatar feltkamptv
feltkamptv - comment - 27 Dec 2017

We already had it working, because we noticed the DELETE query issued by the cron job in the DB. We initialise the Joomla site application and session on the request, but just to be sure we changed it to your method.

We have a lot of cron jobs and background tasks (not through cli) which work on Joomla framework but they do not need to use a session. Being able to disable sessions for some scripts would be a nice to have, to decrease the session inserts to the DB. Something like this:

$params = array('session'=>false);
$mainframe = JFactory::getApplication('site', $params);
avatar mbabker
mbabker - comment - 27 Dec 2017

Some code already exists for that in the application constructor, see

// Enable sessions by default.
if ($this->config->get('session') === null)
{
$this->config->set('session', true);
}
// Set the session default name.
if ($this->config->get('session_name') === null)
{
$this->config->set('session_name', $this->getName());
}
// Create the session if a session name is passed.
if ($this->config->get('session') !== false)
{
$this->loadSession();
}

But, Joomla\CMS\Factory::createSession() isn't aware of this config value so even if the application is told to not initialize the session then the first call to Joomla\CMS\Factory::getSession() will do it arbitrarily. Unfortunately Joomla is heavily coupled to a session so there isn't a way to kill them off in full.

In 4.0, things are a little better. The session service will use a runtime based storage system that doesn't actually create a persistent session for command line environments, so CLI utilities written using the new Console package or at least defining a CLI application environment will not start "real" sessions.

Add a Comment

Login with GitHub to post a comment