shouldn't this c864a01 resolve the opcache issues on update? or is this another issue?

No it doesn't. The thing I reported in the forum I had reproduced using one of my patched update packages, so I tested with updating a 3.5.1 to latest staging (3.5.2-dev), so this change you mentioned was already inlcuded before and after update.

i didn't say it does resolve, i said "shouldn't this" :)

I don't know if it should, maybe @nikosdion knows.

Ah, no, it wouldn't fix that issue because of the order of operations. The correct order of operations should be:

    function finalizeRestore($siteRoot, $restorePath)
    {
        // Clear OPcache
        if (function_exists('opcache_reset'))
        {
            opcache_reset();
        }
        elseif (function_exists('apc_clear_cache'))
        {
            @apc_clear_cache();
        }

        // Make sure Joomla!'s code can figure out which files exist and need be removed
        clearstatcache();

        if (!defined('JPATH_ROOT'))
        {
            define('JPATH_ROOT', $siteRoot);
        }

        $filePath = JPATH_ROOT . '/administrator/components/com_admin/script.php';

        if (file_exists($filePath))
        {
            require_once ($filePath);
        }

        // Remove obsolete files - prevents errors occuring in some system plugins
        if (class_exists('JoomlaInstallerScript'))
        {
            $script = new JoomlaInstallerScript();
            $script->deleteUnexistingFiles();
        }

        // We have modified the filesystem, therefore we must clear the stat cache a second time.
        clearstatcache();

    }

The thing is that if you don't bust the cache before trying to load the script you get the cached script (from the older Joomla! version you are upgrading FROM).

Also, generally speaking, it's a good idea disabling opcode caching before making any code updates. Afterwards clear the cache and re-enable opcode caching. If you think about what opcode caching is supposed to do it makes perfect sense.

@nikosdion You mean the finalizeRestore function should be changed in the core to what you proposed above? If so, who shall do the PR? You? Or me?

Yes, it should be changed in the core. When we wrote that code a few weeks ago we only anticipated opcode problems after the upgrade. I hadn't experienced any issues during the upgrade. Now that you mentioned it I can see exactly why it's happening.

Can you please do the PR yourself? I'm in the middle of deep refactoring my own software, I can't afford to be distracted by core PRs :(

OK, will do PR soon today.

@nikosdion thanks.

Yes, thanks @nikosdion

PR is ##9790 - I will close this issue here as soon as I have tested the PR myself here with success.

Hmm, PR #9790 does not work. @nikosdion can you check?

Let's keep this in one place. I'll close this issue in favour of #9788

@wilsonge I think you meant "... in favour of #9790". #9788 is the one here.
Problem is: #9790 does not help.

Sorry! I blame a complete lack of sleep last night. I'll just close github until tomorrow i think :P

Then have a good night and sleep enough so you are fit tomorrow

re-opened

@mahagr I leave the branch of the closed PR with changes proposed by Nik open, so you can use it for testing: https://github.com/richard67/joomla-cms/archive/fix-jupdate-opcache.zip, custom URL for patched 3.5.2-dev update container is still http://test5.richard-fath.de/list_test8.xml.

In addition I just created a new branch with changes as proposed by you plus the same for restore.php, as I wrote in the closed PR: https://github.com/richard67/joomla-cms/archive/fix-jupdate-opcache-2.zip, custom URL for update test will follow in a while in a comment below.

Meanwhile I will test excluding not the complete com_joomlaupdate folder but only particular files from being cached in ordert to find out which of the php scripts loading causes the error. If I find out something, details will follow also below.

My feeling is that the best would be to suspend (disable) opcaching at all during the update, but am not sure if this can be done in a safe way with later restore of previous opcache setting (enabled or disabled) of the opcache enabled status. Maybe the way would be to save the setting (opcache enabled and similar for apc) in the temporary restoration.php file and restore the setting at the end with the information obtained from that file?

@mahagr Custom URL for a 3.5.2-dev update container based on my new branch mentioned in my comment #9788 (comment) above is http://test5.richard-fath.de/list_test9.xml.

@mahagr For code comparisons here the branch with Michael's changes from the closed PR: staging...richard67:fix-jupdate-opcache.

And here for my new branch with changes as proposed by you plus same in restore.php (was my idea): staging...richard67:fix-jupdate-opcache-2.

But both changesets do not help for my issue.

@mahagr Meanwhile I have found out that it is sufficient to exclude only 1 file from opcache to solve my problem:
administrator/components/com_joomlaupdate/restore.php

If I have only this in the blacklist for opcache so it is not cached, and have deleted cached compiled version of this file, and all other Joomla! files are being cached and compiled versions exist, my issue does not appear anymore, and the update works well.

I will add this info to the forum post mentioned in the issue description, too.

And I will see if I can use your code for this file somehow, and if so and it works, I will make a new PR.

I see that you have done exactly the same in here (the code causes restore.php to recompile):

richard67@c8dee3e

It should work as long as the version you are upgrading from has above code in it.

@mahagr No, the code you proposed causes script.php to recompile, and my extension to your proposal causes restoration.php to recompile. But we have to do it for restore.php, and this is somewhere else.

Then find where the file gets called and hope that its only included/loaded after extracting all files. Adding recompile also for it should do the trick, mostly because of it works without opcache so it gets loaded only after files have been copied over.

The restore.php is not called inside php but via a get or post request or a redirect, I have to check details.

So we have to recompile it before the request/redirect happens.

Question is if we should use the other changes in my commit you referred to, even if they do not help in my case, because they might help for others? Could be related to timings and the revalidation frequency on the particular server for hwich of the files this problem occurs?

Or should we focus on the restore.php only because this is the one with which I consistently can reproduce my problem?

@mahagr There is another place where script.php is called, beside the one where your proposed code takes place:

https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_joomlaupdate/models/default.php#L570

So beside the redirect.php thing which I still am investigating where we could recompile this, that one should also be handled, just to be on the safer side? But if so, there is no check if the file exists in the code up to now, so I would do it a bit different, have the check "if file exists" only around the recompile stuff and then later loading the file is like it is now, i.e. not inside the "if file exists".

Hmm, maybe I should have checked my http log earlier.

Could be that opcache is not the problem but somehow makes the error appear.

I have a 404 not found in my log for the request

POST /administrator/components/com_joomlaupdate/restore.php HTTP/1.1

with the referrer

https://test1.richard-fath.de/administrator/index.php?option=com_joomlaupdate&task=update.install

Note that the post request is not https, and I have in global config secure = admininstrator, means force https for the admin area.

Maybe this is the problem, and not the opcache stuff, and it just works somehow without opcache or when I disable opcache for the restore.php, for whatever reason?

There's no harm of recompiling files that we know to have been changed; in contrary its the only way to make sure the latest version of the file gets loaded. So those recompiles should be added regardless if they fix your issues or not.

You may be right that there's more into this than just recompile. In 3.5.0 I was also experiencing some Joomla caching issues, but I saw another fix for it so I didn't mention it. Maybe the issue you're seeing is also coming from somewhere else..?

@mahagr Well I have meanwhile added that to my new branch you can see here: staging...richard67:fix-jupdate-opcache-2 and have also updated the update zip behind this custom url http://test5.richard-fath.de/list_test9.xml for testing.

I wasn't able to find any time last weekend and I'm quite busy at my work now that my co-worker got back from vacation. I'll try to find some time ASAP, though.

One thing I did notice on my code: opcache_compile_file() causes a warning if opcache had been disabled -- so it's probably better to just remove it and just use opcache_invalidate() on all PHP 5.5+ versions. Someone found the issue in my own code and I've already fixed it -- but it should also be fixed for Joomla changes.

So just use this:

// Compile cached file into bytecode cache
if (function_exists('opcache_invalidate')) {
    opcache_invalidate($file->filename(), true);
} elseif (function_exists('apc_compile_file')) {
    // PHP 5.4
    apc_compile_file($file->filename());
}

APC function doesn't seem to display any warning on failure.

Based on @nikosdion 's comment here #9515 (comment) I close this issue:

the only reasonable solution is asking people to use opcode caching as it was designed to be used: disable it before updating PHP code, re-enable it afterwards.

Yes, that's also what I found when checking Zend Opcache manuals and other sources.

For me my issue seems to be very special for my shared hosting situation, and I have a workaround for it anyway (excluding admin from being cached), and have made this workaround available in the support forum for others who might have the same issue.

@mahagr Feel free to provide a PR if you see a way to solve the issues you had, but for solving my issue all stuff we tried did not help.

Yeah, the issue has proven to be really hard to solve in Joomla. Something still gets cached and I don't know what. I know its possible to have a fix for this, but I'm not sure whats going on.