Tested this patch under the conditions laid out in the original issue and I can confirm it now brings back a proper 404 when before it was not.

@flyingwombats Thanks.
Can you go to https://issues.joomla.org/tracker/joomla-cms/15592 and click on button "Test this" and mark test as success.

I have tested this item ✅ successfully on fb600c2

404 returns correctly

I have tested this item ✅ successfully on fb600c2

Tested and solves the issue, I can confirm it fixes #15647 as well.

RTC after two successful tests.

Sorry, i do not understand quit well what i have to do to fix the issue.

@madesign please read Answer

Why are we creating files using the c flag? Isn't the r good enough?
For your information, the function register_shutdown_function is not 100% reliable

Why are we creating files using the c flag? Isn't the r good enough?

r does not create a file. No file - no lock.
I have to create a lockfile before I will create a cache.

For your information, the function register_shutdown_function is not 100% reliable

Function register_shutdown_function should be called in almost 100% cases, except fatal errors.
I know it looks weird but callback function can terminate process by exit(0).

May be the best way would be to not block other processes to create a cache data but only block on save the data to cache file.

Old version does not create a lock for not existed cache files.
There was an error because for example: 10 processes can create the same cache file in the same time and then they override the file a few times.

I've just tried this fix on PHP 5.5.9 and it is now not saving any page cache files. I was suffering from the "zero length cache file" problem for pages which the page cache plugin ignored. This fix is now removing all cache files.

We're using a custom page cache plugin based on the Joomla one. It was working fine on 3.6.x. I'll also check that there aren't any changes to the 3.7 page cache plugin we should be considering.

IMO your problem is somewhere else.
Take a look at https://github.com/joomla/joomla-cms/blob/staging/libraries/joomla/cache/cache.php#L251-L254

I think about redesign it once again and remove cache locking from outside get and store methods.

Can you put fflush($_fileopen); after $result = @fwrite($_fileopen, $data, $length); line 216 in file libraries/joomla/cache/storage/file.php and check if this resolve your problem?

OK. Do you mean in the patched file.php or the Joomla 3.7.0 version?

In the patched file

You can also add a few lines below fclose($_fileopen) another function clearstatcache();

I've got the following:

                if ($_fileopen)
                {
                        $length = strlen($data);
                        $result = @fwrite($_fileopen, $data, $length);
                        fflush($_fileopen);

                        if ($close)
                        {
                                @fclose($_fileopen);
                        }
                        clearstatcache();

                        return $result === $length;
                }

But it's not helping. There are still no files ending up in cache
On a different server (same site, same OS - Ubuntu 14.04) it's working OK. Could we have a race condition?

Example is OK. Then I have no more ideas now.
One php process should create some cache files. Do you have correct permission and free space on disk?

OK, it's working now. It looks like it was trouble at my end which occurred at the exact same time I applied this patch.

It seems that the cache/page folder was recreated in some way, probably via an rsync action, while my terminal's CWD was in that folder. So, when I ls -l I see no cache files. However, several hours later I change directory and voilà, all the files are in the 'new' cache/page folder and it was working all along.

Sorry to waste your time and thank you for your efforts to sort this out.