? Success

User tests: Successful: Unsuccessful:

avatar csthomas
csthomas
20 May 2016

Purpose

I want to fix a few bugs in memcache structure:

  • stop memcache extension to always serialize/unserialize whole cache keys in one $index var
  • split one locked index to more separated indexes (each per group_key)
  • change index keys from auto increment long integer to keys which are group_keys. After a long running time auto increment key may exceed max long integer limit.

Summary of Changes

Index structure before:

array(
    stdobject(name="[secret]-cache-[com-content]-[cache_id]", size=1024),
    stdobject(name="[secret]-cache-[com-content]-[cache_id]", size=1024),
    stdobject(name="[secret]-cache-[com-content]-[cache_id]", size=1024),
    stdobject(name="[secret]-cache-[com-content]-[cache_id]", size=1024),
    ...
)

Index structure after:
Now index is separated into more arrays ($index and $index2 per group key)

$index is array of group_keys. Example: array("com_contentE", "com_contentF");

$index = array(
    [_system],
    [com_contentA],
    [com_contentB],
    [com_contentC],
    ...,
    [com_contentE],
    [com_contentF],
    ...
);

// index2 has address at "[secret]-[group_key]-index]"
$index2 = array(
    "[com_content]-[md5sum_of_cache]" => 1025, 
    "[com_content]-[md5sum_of_cache]" => 1325, 
    "[com_content]-[md5sum_of_cache]" => 1425, 
    ...
);

Instead of 1 index we will have 1 + total group_keys indexes and then lock time won't be a problem.

This patch contains some debug staff.

You can see log debug in logs/memcache.php after you click on "Clear Expired Cache" on backend.

Test instruction

In progress. Please code review before.

avatar csthomas csthomas - open - 20 May 2016
avatar csthomas csthomas - change - 20 May 2016
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 20 May 2016
Labels Added: ?
avatar csthomas csthomas - change - 20 May 2016
The description was changed
avatar csthomas csthomas - change - 21 May 2016
The description was changed
avatar csthomas csthomas - change - 21 May 2016
The description was changed
avatar csthomas csthomas - change - 21 May 2016
The description was changed
avatar brianteeman brianteeman - change - 21 May 2016
Category Cache
avatar mbabker
mbabker - comment - 22 May 2016

As a high level API class, the cache storage adapters shouldn't be tied to
the application classes or making decisions on whether a message gets
enqueued for display or not. The adapter should simply fail and a lower
level class should be handling that error. So yes, the adapter should be
throwing an Exception and only that. Tying it to the application classes
creates a very deep level of coupling to those specific implementations and
the static global factory, both of which should be minimized as practical
in the high level library classes.

On Sunday, May 22, 2016, Tomasz Narloch notifications@github.com wrote:

I will comment more later. Currently I remove deprecated stuff.

For current version of joomla I can not agree that I should revert to:
throw new RuntimeException('Could not connect to memcache server');
so I use:
$app->enqueueMessage('Could not connect to memcache server', 'error');

When I reverted to throw exception and memcached server failed then I
could not login to backend.
With enqueueMessage I can and I see only error.
(The main result about success connection is boolean value from return)

I think that when memcached server failed then joomla should still
working.
This is not major part of joomla that shut down all backend or frontend.

Admin should have possibility to login and fix the problem in
configuration or restart memcached server after he see error message.
Joomla will be only slower for time when memcached server is down.

About isSupport() method

When admin want to use memcached from not standard address (ex. other
machine, unix socket, other port) then on joomla <= 3.5.1 admin can not see
field to change host and port on global config.
This is silly.

I know that there is a workaround. Admin has to start memcached server on
standard local address (localhost:11211) and then change to some other
address, but this is not a solution.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#10565 (comment)

avatar csthomas
csthomas - comment - 23 May 2016

I have reverted to throw exception.

avatar csthomas
csthomas - comment - 25 May 2016

Cache index in memcache storage of joomla 3.5
Example cache id:

$cache_id = "11112222333344445555666677778888-cache-_system-429adb9b368342c818ec393fb9780000"

JOOMLA 3.5 index structure:

$index = array(
    stdClass($name='11112222333344445555666677778888-cache-_system-429adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-com_content-029adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-com_content-039adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-com_content-349adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-com_content-329adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-mod_menu-139adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-mod_menu-149adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-mod_menu-229adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-mod_menu-239adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-mod_menu-249adb9b368342c818ec393fb9780000', $size=1024),
    stdClass($name='11112222333344445555666677778888-cache-mod_menu-259adb9b368342c818ec393fb9780000', $size=1024),
    ...
    stdClass($name='11112222333344445555666677778888-cache-mod_menu-269adb9b368342c818ec393fb9780000', $size=1024),
);

Serialize and unserialize $index for methods like get/remove/clean/store takes a lots of time.
Array keys are numbers (long int) and may overflow for long running memcache/php.

Cache index in the new memcache storage will be split into more indexes

Primary $index and secondary indexes $index2 (per group key).

Group key is a combination of group name ex. 'com_content' and first char (hex value 0-9a-f) from md5sum or only group name otherwise:

11112222333344445555666677778888-cache-mod_menu-269adb9b368342c818ec393fb9780000
11112222333344445555666677778888-cache-_system-429adb9b368342c818ec393fb9780000

For technical reason I have to replaced group key 'mod_menu2' to 'mod_menuC'.

Group key is generated by code below:

        $key = $group;

        // If the group does not begin with '_' and does not end with number
        // then add suffix A or B or ... or P.
        // For groups like _system or com_custom_part1 do not add suffix
        if ($group[0] !== '_' && !ctype_digit(substr($group, -1, 1)))
        {
            // The cache id suffix looks like:
            // com_content-329adb1b3633424818ec393fb9780000 and then key will be com_contentD
            // Total can be 16 different keys per one group
            $key .= chr(hexdec($cache_id_sfx[strlen($group) + 1]) + 65);
        }

        return $key;

Now primary index is:

$index = array(
    '_ssytem',
    'com_contentA',
    'com_contentD',
    'mod_menuB',
    'mod_menuC',
    ...
);

and secondary $index2 (per group key) looks like this example for 'mod_menuC'

//$index2 = $memcache→get('11112222333344445555666677778888-mod_menuC-index');
$index2 = array(
    'mod_menu-229adb9b368342c818ec393fb9780000' => 1024,
    'mod_menu-239adb9b368342c818ec393fb9780000' => 1024,
    'mod_menu-249adb9b368342c818ec393fb9780000' => 1024,
    'mod_menu-259adb9b368342c818ec393fb9780000' => 1024,
);

It more looks like a tree structure, where index is parent and indexes of index2 are leafs.

avatar csthomas
csthomas - comment - 29 May 2016

I have plan to remove some debug logs and may be memcache log file at all.
Error/warning logs will be changed a little.

I want to stay with debug logs for a testing and better understanding current changes.
If there is still issue with my description may be should I create some graph about cache indexing?

avatar csthomas csthomas - change - 30 May 2016
Title
[cache] First proposal of changing structure of memcache data
[memcache] Split one array of memcache index to more indexes in order to better performance
avatar csthomas csthomas - change - 30 May 2016
The description was changed
avatar csthomas
csthomas - comment - 1 Jun 2016

Error logs in logs/memcache.php look like

Saving data failed for com_jmapA (size: 2538849, compression: 0)

Which means that memcached server can not store values bigger that 1MB, but when we enable compression then memcache store that variable.

Lock index2 failed for mod_menuK after 0.037997

Timeout after 0.037997 second, server is very busy

Total index2 arrays: 172, max full index2 was com_contentK: 127
Total expired: 1198, max saving time in last 24h: 0.109072

This became from (gc function) button "Clear expired cache".
There is 172 arrays of index2, the biggest array of index2 contains 127 items.
Gc function removed from index2 arrays 1198 items and max time of store cache takes 0.109072 second.

avatar joomla-cms-bot joomla-cms-bot - change - 30 Aug 2016
Category Cache Libraries Cache
avatar joomla-cms-bot joomla-cms-bot - change - 30 Aug 2016
Labels Added: ?
avatar csthomas csthomas - change - 30 Aug 2016
Labels Added: ?
avatar csthomas
csthomas - comment - 28 Jun 2017

As memcache is not used for php 7.x and
I have no more interest with it I close it.

avatar csthomas csthomas - change - 28 Jun 2017
Status Pending Closed
Closed_Date 0000-00-00 00:00:00 2017-06-28 10:59:51
Closed_By csthomas
avatar csthomas csthomas - close - 28 Jun 2017

Add a Comment

Login with GitHub to post a comment