? ? Success

User tests: Successful: Unsuccessful:

avatar csthomas
csthomas
9 Jun 2016

Pull Request for Issue #10706.

Summary of Changes

  1. Eliminate usleep(100) when cache file does not exist.
  2. Do not call flock() if the file does not exist, replace flag r+b to c+b in fopen function.
  3. Do not allow to release lock, remember $_fileopen resource in $this->_locked_files variable.
  4. At unlock() do not re-open cache file, but release lock only.

Testing Instructions

Code review.
Or standard test below.

Before patch.

  1. Enable cache and set cache handler to File,
  2. Replace line 309 in libraries/joomla/cache/storage/file.php ( https://github.com/joomla/joomla-cms/pull/10767/files#diff-056f1b3f5ee42d2b80e891d375789137L309 )
    from:
usleep(100);

to

usleep(100); JLog::add("Joomla call usleep!!!", JLog::WARNING, 'jerror');
  1. Remove all files from cache. Clear cache.
  2. Now go to front-end.
  3. You should see "Joomla call usleep!!!"

After patch do the same but
at point 5 you should not see "Joomla call usleep!!!"

avatar csthomas csthomas - open - 9 Jun 2016
avatar csthomas csthomas - change - 9 Jun 2016
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 9 Jun 2016
Labels Added: ?
avatar brianteeman brianteeman - change - 10 Jun 2016
Category Cache
avatar brianteeman
brianteeman - comment - 10 Jun 2016

Without any testing instructions this is unlikely to get tested very quickly


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/10767.

avatar csthomas
csthomas - comment - 10 Jun 2016

I have added a test instruction in first comment.

avatar brianteeman
brianteeman - comment - 10 Jun 2016

Thank you

avatar csthomas csthomas - change - 17 Jun 2016
The description was changed
avatar csthomas csthomas - change - 24 Jun 2016
The description was changed
avatar csthomas csthomas - change - 24 Jun 2016
Title
[cache] File storage - create file before flock
[cache file] Create file before call flock()
avatar csthomas csthomas - change - 24 Jun 2016
Title
[cache] File storage - create file before flock
[cache file] Create file before call flock()
avatar csthomas csthomas - change - 9 Jul 2016
Title
[cache file] Create file before call flock()
[cache file] Improve performance - eliminate the first usleep, do not try to lock nonexistent file
avatar joomla-cms-bot joomla-cms-bot - change - 23 Aug 2016
Category Cache Libraries Cache
avatar csthomas csthomas - change - 23 Aug 2016
The description was changed
avatar csthomas csthomas - edited - 23 Aug 2016
avatar csthomas csthomas - edited - 23 Aug 2016
avatar csthomas csthomas - change - 23 Aug 2016
The description was changed
avatar csthomas csthomas - edited - 23 Aug 2016
avatar csthomas csthomas - change - 24 Aug 2016
Title
[cache file] Improve performance - eliminate the first usleep, do not try to lock nonexistent file
[cache] File handler - Improve performance
avatar csthomas csthomas - edited - 24 Aug 2016
avatar csthomas csthomas - change - 31 Aug 2016
The description was changed
avatar csthomas csthomas - edited - 31 Aug 2016
avatar joomla-cms-bot joomla-cms-bot - change - 1 Sep 2016
Category Cache Libraries Unit Tests Cache
avatar joomla-cms-bot joomla-cms-bot - change - 1 Sep 2016
Labels Added: ?
avatar csthomas csthomas - change - 1 Sep 2016
The description was changed
avatar csthomas csthomas - change - 1 Sep 2016
The description was changed
avatar csthomas csthomas - change - 4 Sep 2016
The description was changed
avatar csthomas
csthomas - comment - 5 Sep 2016

Feel free to test.

avatar csthomas csthomas - change - 20 Sep 2016
The description was changed
avatar franz-wohlkoenig
franz-wohlkoenig - comment - 8 Jan 2017

I have tested this item successfully on c41eda0


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/10767.

avatar franz-wohlkoenig franz-wohlkoenig - test_item - 8 Jan 2017 - Tested successfully
avatar joomla-cms-bot joomla-cms-bot - change - 26 Jan 2017
Category Cache Unit Tests Libraries Unit Tests Cache
avatar csthomas csthomas - change - 26 Jan 2017
The description was changed
avatar csthomas csthomas - edited - 26 Jan 2017
avatar csthomas csthomas - change - 26 Jan 2017
Title
[cache] Skip usleep(100) in file handler, fix lock() bug
Remove uslep(100) from first call cache in file handler
avatar csthomas csthomas - edited - 26 Jan 2017
avatar csthomas csthomas - change - 26 Jan 2017
The description was changed
avatar csthomas csthomas - edited - 26 Jan 2017
avatar csthomas csthomas - change - 26 Jan 2017
Title
Remove uslep(100) from first call cache in file handler
Remove uslep(100) from first call of cache->get in file handler
avatar csthomas csthomas - edited - 26 Jan 2017
avatar csthomas csthomas - change - 26 Jan 2017
The description was changed
avatar csthomas csthomas - edited - 26 Jan 2017
avatar csthomas csthomas - edited - 26 Jan 2017
avatar csthomas csthomas - change - 31 Jan 2017
The description was changed
avatar csthomas csthomas - edited - 31 Jan 2017
avatar marrouchi
marrouchi - comment - 1 Feb 2017

I have tested this item successfully on b85ddca


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/10767.

avatar marrouchi marrouchi - test_item - 1 Feb 2017 - Tested successfully
avatar csthomas
csthomas - comment - 1 Feb 2017

Thanks for tests.

First test was done before 'resolve merge conflict' but 'conflicted' merged PR 13372 only change " to '.
Can we add RTC as we have two success tests.

avatar jeckodevelopment
jeckodevelopment - comment - 3 Feb 2017

@csthomas We should have another test, please, since we had new commits after the test of @franz-wohlkoenig.
If he can test it again, it would be great

avatar franz-wohlkoenig
franz-wohlkoenig - comment - 3 Feb 2017

will do

avatar franz-wohlkoenig
franz-wohlkoenig - comment - 3 Feb 2017

I have tested this item successfully on b85ddca


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/10767.

avatar franz-wohlkoenig franz-wohlkoenig - test_item - 3 Feb 2017 - Tested successfully
avatar csthomas
csthomas - comment - 3 Feb 2017

thanks

avatar dgt41
dgt41 - comment - 3 Feb 2017

I have tested this item successfully on b85ddca


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/10767.

avatar dgt41 dgt41 - change - 3 Feb 2017
Status Pending Ready to Commit
avatar dgt41 dgt41 - test_item - 3 Feb 2017 - Tested successfully
avatar dgt41
dgt41 - comment - 3 Feb 2017

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/10767.

avatar rdeutz rdeutz - change - 4 Feb 2017
Status Ready to Commit Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2017-02-04 20:39:00
Closed_By rdeutz
Labels Added: ?
avatar rdeutz rdeutz - close - 4 Feb 2017
avatar rdeutz rdeutz - merge - 4 Feb 2017
avatar wilsonge
wilsonge - comment - 25 Apr 2017

Changing r+b to c+b appears to be responsible for #15544

avatar csthomas
csthomas - comment - 25 Apr 2017

I'm checking it

avatar wilsonge
wilsonge - comment - 25 Apr 2017

Thanks :)

avatar csthomas
csthomas - comment - 26 Apr 2017

I see a few solution.
The reason is that the cache create a lock before call view method.
If view method generete 404 then process call error page and die.
Cache method does not get any response about failed view.
Cache file is empty and stay to the next requests.

  1. First solution is dirty, because because use registered shutdown function to delete such cache file.
diff --git a/libraries/joomla/cache/storage/file.php b/libraries/joomla/cache/storage/file.php
index f2dba86e5f..c4d01e4ed7 100644
--- a/libraries/joomla/cache/storage/file.php
+++ b/libraries/joomla/cache/storage/file.php
@@ -361,6 +361,19 @@ class JCacheStorageFile extends JCacheStorage
                {
                        // Remember resource, flock release lock if you unset/close resource
                        $this->_locked_files[$path] = $_fileopen;
+
+                       $pointer =& $this->_locked_files[$path];
+
+                       // Delete file on unexpected die.
+                       $deleteMe = function () use ($pointer, $path)
+                       {
+                               if ($pointer)
+                               {
+                                       unlink($path);
+                               }
+                       };
+
+                       register_shutdown_function($deleteMe);
                }
  1. Second solution is simple, does not delete such file but only check file size:
diff --git a/libraries/joomla/cache/storage/file.php b/libraries/joomla/cache/storage/file.php
index f2dba86e5f..ac0f54ccad 100644
--- a/libraries/joomla/cache/storage/file.php
+++ b/libraries/joomla/cache/storage/file.php
@@ -413,7 +413,7 @@ class JCacheStorageFile extends JCacheStorage
                {
                        $time = @filemtime($path);
 
-                       if (($time + $this->_lifetime) < $this->_now || empty($time))
+                       if (($time + $this->_lifetime) < $this->_now || empty($time) || filesize($path) === 0)
                        {
                                @unlink($path);

I will create a PR for above option 2.

Other solutions may require to catch each error as exception in the cache method and then delete the cache file or try to cache error response.

avatar demis-palma
demis-palma - comment - 28 Apr 2017

Hi @csthomas I have experienced another issue due to 'r+b' -> 'c+b' replacement.
Before this change, for a module included in an article with {loadmodule}, it was possible to call JCache::setCaching(false) to disable caching of the container article.
Now an empty file remains in the directory instead, which prevents subsequent view of the article until the cache expires.

Therefore I ask myself if 'c+b' is really necessary? What's the advantage of creating an empty file when you still don't need it?
Otherwise I'd suggest to restore the previous mode 'r+b'.

avatar csthomas
csthomas - comment - 28 Apr 2017

About creating an empty file.

  • The empty file is required to lock this file by flock(), to prevent other php processes to create the same cache file in the same time.

About your situation I understand that:

  1. The cache is enabled.
  2. There is a request for article view.
  3. The cache library tries to find a cache file but there is not file.
  4. The cache library creates an empty file and locks it.
  5. The cache library calls $view->display()
  6. The module loaded by {loadmodule} disables the cache by JCache::setCaching(false)
  7. The cache library gets the data but does not store it.
  8. The empty cache file left.
  9. At the next request the empty cache file exists and there is the problem.

Take a look at #15592 it should resolve your problem.

avatar demis-palma
demis-palma - comment - 29 Apr 2017

@csthomas your description is so accurate that for a second, I was afraid that you control my PC with a malware. ?

I understand the need for file locking, but this way we have changed so deeply the behaviour of the file cache, that the problem reported up to now are just the first in a long line that I'm sure will be reported.

I think that the current implementation has a major architectural problem that was already there before your PR, and that can be resumed in "do not lock for longer than is strictly necessary".
Since we currently:

  1. lock the file 2 hours and a halt before we are ready to write data into it, and
  2. give controls to third party components and modules in the meantime,

then it is reasonable to expect that something will go wrong.
Your attempt to fix it using the shutdown function is clever, but as you said, it's dirty.

To me it was badly designed before your PR, but the 'r+b' mode did prevent early lock.
It's one of those cases where two errors compensates each other.
Having changed the mode to 'c+b' has highlighted the problem.

I just wanted to know if you are agree with that, and I'll start a full refactoring.

Add a Comment

Login with GitHub to post a comment