@nikosdion

I haven't tested upgrade from 4.1.5 but I tried upgrade from 4.0.4, 4.1.0 to 4.2.0 success without this PR.

I tried to upgrade from 4.0.0 to the package generated by this PR but still get the original error. So for some reasons, it does not solve the upgrade issue yet. Not sure if I missed something with the testing.

@zero-24 Back in Joomla 3.4 we had a similar problem where you could not update Joomla unless Joomla Update was updated first. At that point Joomla Update became an updateable extension to overcome this problem. Instead of making a full CMS release for a no longer maintained CMS version we can simply release a new version of Joomla Update, using the feature we already have in place for snafus like the one we are experiencing here.

I would not issue an new core release but use the first that was using the new way and worked in that point of time. As we had only issue reported now on very old versions. I would like to try to avoid issuing an standalone joomla update release which spans over so many joomla releases to avoid other issues we might get. So the plan would be 4.0.0 -> upgrade to a version that has the new updater code which support finalization -> got to 4.2.0 where this patch here comes into play and makes sure the autoload file is removed.

@nikosdion Thanks for the clear instructions. So the 4.1.5 -> 4.2.0 worked fine. The autoload_psr4.php was re-created. Going to run the 4.0.0 test now.

@zero-24 If you upgrade 4.0.0 to 4.0.4 or later 4.0.x release then you can safely upgrade to 4.2.0 with this PR. However! You can't do that... The only proposed update you are given by Joomla itself on a Joomla 4.0.0 site is 4.2.0. Therefore we are guiding people to an one-click breakage of their site.

The only viable solution is releasing a new version of JUST Joomla Update. Not the entire CMS. This will also address the problems people like me had with 4.1.5 to 4.2.0 updates. It's a win-win situation.

@nikosdion I replaced the complete administrator/components/com_joomlaupdate from the package built by this PR and placed it in the 4.0.0 site. When I go to the Joomla Updater I see this error:
Call to undefined method Joomla\Component\Joomlaupdate\Administrator\Model\UpdateModel::getDatabase() which comes from line 98 $db = $this->getDatabase(); and I believe this is the new code for 4.2.0.

Can you check please?

Yeah, that was changed very recently. Let me see if I can convince Joomla Update to run on Joomla 4 with some judicious use of if-blocks.

@zero-24 If you upgrade 4.0.0 to 4.0.4 or later 4.0.x release then you can safely upgrade to 4.2.0 with this PR. However! You can't do that... The only proposed update you are given by Joomla itself on a Joomla 4.0.0 site is 4.2.0. Therefore we are guiding people to an one-click breakage of their site.

Yes that can be changed right now we only propose the update from 4.0.0 to 4.1.5 will prepare a PR to update the update server and point them to go via 4.0.4 first so we dont add more steps for that people affected by this.

@zero-24 There were some reports a few months ago that 4.0 to 4.1 was causing the same problems for the same reason. Let's try to get the new com_joomlaupdate running on Joomla 4.0.0–4.0.4 instead.

@zero-24 There were some reports a few months ago that 4.0 to 4.1 was causing the same problems for the same reason. Let's try to get the new com_joomlaupdate running on Joomla 4.0.0–4.0.4 instead.

4.0.what to 4.1? Did you found the reports? And wouldnt that kind of issues be solved by this PR? The change to the update distribution is only making sure everyone uses the new updater.

This is what i had in mind: joomla/update.joomla.org#263 (comment)

I have updated this PR so that Joomla Update can now safely run on Joomla 4.0.0.

I have confirmed that installing the new Joomla Update (replacing administrator/components/com_joomlaupdate, media/com_joomlaupdate and administrator/language/en-GB/com_joomlaupdate*) on a Joomla 4.0.0 site works AND proposes an upgrade to 4.1.5.

Installing the upgrade to 4.1.5 DOES update the autoload_psr4.php file.

On that site, I did the same Joomla Update upgrade but it does not see the Joomla 4.2 version. No idea why. Huh. I did a manual update (Upload & Install) to Joomla 4.2.0. This worked and the upgrade to 4.2.0 DOES update the autoload_psr4.php file.

On that site, I did the same Joomla Update upgrade but it does not see the Joomla 4.2 version. No idea why. Huh. I did a manual update (Upload & Install) to Joomla 4.2.0. This worked and the upgrade to 4.2.0 DOES update the autoload_psr4.php file.

Yesterday we have pulled 4.2.0 off from the update server thats the reason your site dont see it via the regular update path. :)

@zero-24 Ah, good to know. So, I tested 4.0.0 to 4.2.0 after installing com_joomlaupdate from this repository (using Upload & Install, of course) and it works correctly. The autoload_psr4.php file is replaced, therefore it fixes this issue.

I just need a couple of tests and you can merge this PR and release a new standalone com_joomlaupdate update to solve all of the weird upgrade problems which ultimately are caused by autoload_psr4.php not being updated.

Post Scriptum: I am very wary of the idea that the autoload_psr4.php file is never updated unless manually deleted. What I'd like to do is have a Global Configuration option with its update frequency in minutes. Three possible defaults are 525600 (one year; suitable for production sites operated by Those Who Know What They Are Doing), 360 (6 hours; not too bad for production, not ideal either) and 1 (development mode). I would be inclined to recommend a default of 360 minutes. I have been burned too much by this file not being updated and there's no documentation I could find about it. If that sounds like something you guys would be willing to test and back I am willing to write the code.

The update from 4.0.0 to 4.2.0 worked here as well however we can have the following update path from the update servers, so users do not have to update com_joomlaupdate:

Update sequence from 3.10.x
3.10.x -> 4.0.4
4.0.4 -> 4.2.1

Update sequence from 4.0.0
4.0.0 -> 4.0.4
4.0.4 -> 4.2.1

Update sequence from 4.1.5
4.1.5 -> 4.2.1

My doubt is now if we should or should not offer the com_joomlaupdate update as well.

Post Scriptum: I am very wary of the idea that the autoload_psr4.php file is never updated unless manually deleted. What I'd like to do is have a Global Configuration option with its update frequency in minutes. Three possible defaults are 525600 (one year; suitable for production sites operated by Those Who Know What They Are Doing), 360 (6 hours; not too bad for production, not ideal either) and 1 (development mode). I would be inclined to recommend a default of 360 minutes. I have been burned too much by this file not being updated and there's no documentation I could find about it. If that sounds like something you guys would be willing to test and back I am willing to write the code.

Hmm It was also updated when you installed or updated extensions. Not sure whether its worth doing it when no changes to the classes are done just on a regular basis and would that maybe even cause more issues than it solves? It is also removed as soon as you clear the cache folder. I would say lets branch that out of this PR into a dedicated issue and lets discuss that there before you have to start writing the code.

@roland-d Yes, we must issue a com_joomlaupdate update as well. Otherwise the 4.0.4 to 4.2.1 MIGHT not work. I checked the site where all hell broke loose yesterday. Apparently the finalise.php script failed to execute completely. I had schema problems AND a stale autoload_psr4.php file. I am still trying to bring this site back to life (I refuse to restore it from backup and retry, I want to understand what the heck is going on).

@zero-24 Agreed. I will start a discussion when I am done working on the issues I've bumped on since last night. I have horror stories to share from extension updates as well ?

@roland-d Yes, we must issue a com_joomlaupdate update as well. Otherwise the 4.0.4 to 4.2.1 MIGHT not work

That would mean we have to check any changes to com_joomlaupdate made between 4.0.0 and 4.2.0 and all supporting code and verify that they are working on 4.0.0 too. For example we would need a similiar version compare about "twofactor" vs "mfa" notjing critical but it should be checked whether there are other changes.

Until now we only did standalone releases that for one specific release when nothing else works, First I would like to understand why on some sites it might result into an error or not. We have not got any reports about issues until now.

Well, I AM reporting an issue! As I said two comments above:

I checked the site where all hell broke loose yesterday. Apparently the finalise.php script failed to execute completely. I had schema problems AND a stale autoload_psr4.php file.

The site is thankfully something I am using for development but on a live, commercial VPS hosted with Rochen. That was a Joomla 4.0.3 site.

You know what is crazy? That site is practically empty! It has Joomla and SocialLogin. I am using it to develop SocialLogin plugins since most services require a live site to give me a developer account.

My business site is far more complex and it updated just fine, though it was FROM 4.1.5 TO 4.2.0 — naturally, that site is kept fully up-to-date.

I am also discovering other weird issues with WebAuthn on the special snowflake site... but that's for another issue I am composing right now.

4.0.3 to 4.2.0 is confirmed to fail as that is running the outdated code of joomla update. would be interesting to know whether we have still issues using a upgrade package with this patch applied here from anything between 4.0.4 and 4.1.

@zero-24 Ah, damn, I found out what happened on that special snowflake site — thank you for being my rubber ducky. Remember how I told you that I am using it to develop SocialLogin? That came with an old version of the JWT library. Joomla has a newer version. In his infinite wisdom, the library developer changed the data type of the return claims when decoding the JWT from an array of arrays to an array of objects. Since my plugin was loading the old version (returning an array of arrays) it caused plg_system_webauthn to fail because it expects an array of objects. This seems to have caused the finalisation code to fail. I did not catch the problem because the early failure of the finalisation code also means that old files were not removed, the schema was not updated and yes, the autoload_psr4.php was not removed (since upgrading from 4.0.3 won't do that, right...).

So... this is actually a bit troubling. If a system plugin fails on 4.2 we have quite the problem. While extract.php will do its job of extracting the archive and running finalise.php, running the task=update.finalise will fail.

The only solution I can possibly think is having yet another directly web accessible file with a custom WebApplication which does NOT load any plugins to run the code we currently have in task=update.finalise. Otherwise all it takes to have a broken update (and a broken site) is a system plugin which fails to work correctly.

The only solution I can possibly think is having yet another directly web accessible file with a custom WebApplication which does NOT load any plugins to run the code we currently have in task=update.finalise. Otherwise all it takes to have a broken update (and a broken site) is a system plugin which fails to work correctly.

Is it possible to add a feature to "unload" the plugins before that step and load them again after that? I'm not sure whether another directly to call file is a good solution. When its the only solution we have to go that route but maybe we put some thougth into whether we can somehow avoid it?

If anything that broken system plugin would have caused issues on any update right?

If anything that broken system plugin would have caused issues on any update right?

Absolutely correct.

Is it possible to add a feature to "unload" the plugins before that step and load them again after that?

I love this idea! Give me an hour or two. I need some coffee, and to tidy up the house while I am thinking this through (doing boring, manual labor helps me think).

just logged in to suggest the same thing. should help with updates from 3 with problem plugins as well

I came up with a solution.

I introduced the PluginFreezingApplicationInterface which is only implemented by the AdministratorApplication and only has one public method to inspect the state of the plugin loading freeze flag.

The PluginHelper checks if the application implements this interface and inspects the flag. If the flag is set it will refuse to load any plugins except for a small selection of core plugins necessary to execute the upgrade finalisation and all plugins under the installer folder. For the latter plugins I added try/catch statements which will prevent any error during their loading or execution from bringing down the application as this would make the post-update cleanup fail.

The AdministratorApplication sets the freeze flag if and only if we are in one of the concerned Joomla Update states:

• The finaliseconfirm layout which asks the user to “log in” (verify their username and password) if the session expired while the update was being extracted
• The finaliseconfirm task which is used to verify this “log in”
• The update.finalise task which performs the post-extraction update finalisation
• The update.cleanup task which cleans up the leftover files after the update and delivers an operable site.

If a system plugin fails to execute your site will still be broken BUT it will be fully updated.

As @brianteeman correctly pointed out, this change also addresses the problem of updates from Joomla 3 to Joomla 4 if there are always loading plugins (system, behaviour, actionlogs, user, authentication, maybe something I forget off the top of my head). This has been a major pain in the posterior of people doing updates. At least they will have a correctly updated site so they can start renaming plugins / editing the #__extensions table to get it back to an operable state.

My brain is mush, I'd appreciate it if someone came with a good testing plan for this.

@nikosdion Why is this PR deleting the old "restore_finalisation.php" file? Don't we need that when updating older versions because the update component's old JS is still loaded in the browser, and this will try to call this old "restore_finalisation.php"?

And why is this PR deleting the "pkg_search.xml"? We need to ship this file not with new iallations but with updates when updating from a version where the old search is still present.

@richard67 Shoot, I fat-fingered the commit. Let me fix that.

@richard67 I fixed that and I updated .gitignore so next time I accidentally delete pkg_search.xml it will scream at me. I can't guarantee I will hear it screaming but at least it will be telling me I am removing a tracked file :p

@richard67 I fixed that and I updated .gitignore so next time I accidentally delete pkg_search.xml it will scream at me. I can't guarantee I will hear it screaming but at least it will be telling me I am removing a tracked file :p

@nikosdion You might do that with your .gitignore locally, but the .gitignore file is not ignored itself, so it is part of the source code, and your PR now adds the "pkg_search.xml" file to the .gitignore in the sources. I don't think that's right.

Yeah, but pkg_search.xml was ALREADY in the repo despite being ignored and you asked me why I removed it.

So one of the two is wrong.

A. The file is in the repository.

B. The file is not in the .gitignore

Which one is wrong so I know what to fix?

Yeah, but pkg_search.xml was ALREADY in the repo despite being ignored and you asked me why I removed it. So one of the two is wrong. A. The file is in the repository. B. The file is not in the .gitignore Which one is wrong so I know what to fix?

The file is normally not touched, but when you are testing updating on a git clone, it is removed at the end of the update if search is not installed. So I think it is right as it is, the file in the sources and not in the .gitignore.

But maybe I am wrong.

@wilsonge What do you think?

Meanwhile I’ve noticed the exclamation mark and the beginning of that line. So the .gitignore change should be ok.

My 2 cents, it is fine to track this file, so we know if it accidentally gets modified.

My brain is mush, I'd appreciate it if someone came with a good testing plan for this.

I would think we need something like this:

• A plugin that is bad installed on a site to be updated
• Check that the update fails
• Apply this patch
• Check that the update succeeds

I only do not know what the bag plugin criteria should be. Is it just a system plugin that dies or returns false somewhere? Which event should it be triggered on?

I only do not know what the bag plugin criteria should be. Is it just a system plugin that dies or returns false somewhere? Which event should it be triggered on?

I would say a system plugin with an error in the constructor could do the trick. As that was an issue with all releases where we have joomlaupdate aviable already I would say we should branch out that freezing plugins patch into a dedicated PR against 4.2.2 so we can get more tests in but get the 4.2.1 release out early too.

What about to remove all listeners from the dispatcher during upgrade and then reinstate them again after the process is done?

Yeah, but pkg_search.xml was ALREADY in the repo despite being ignored and you asked me why I removed it.

So one of the two is wrong.

A. The file is in the repository.

B. The file is not in the .gitignore

Which one is wrong so I know what to fix?

The file is supposed to be in the repo for people upgrading from j3 who need the update server definition to be moved to the external github page. for people starting new sites on j4 it's junk and gets removed as part of installation. In theory it's fine to add it to gitignore as long as nothing changes in the future. just in the off chance anything is intentionally changed - not sure how many of our contributors will know about force git adding

@nikosdion Do you have time to finish up this PR today? The reason I am asking is because I would like to push out the RC today.

As Tobias said, it will be good to move the plugin unloading to a separate PR as this still needs more thinking/testing/coding. Could you move that to a new PR?

With the plugin unloading out of this PR, it should be ready to do the final test and merge.

@wilsonge If something needs to change in the pkg_search.xml I am sure we can find a seasoned contributor to do or help with that :)

@laoneo The problem is not just dispatching events but also creating the plugin object. For example, my plugin's constructor was loading the autoload.php of a private Composer vendor directory which had an older version of the same JWT library we ow have, in a newer version, in Joomla itself. The plugin was an integration with the Login with Apple service which uses JWT. Joomla 4.0 and 4.1 needed the extra library as it wasn't available in Joomla at the time. The old version returned an array of arrays (array[]) for the JWT claims, the new version returns an array of objects (stdClass[]) for the JWT claims. The WebAuthn system plugin has an array_map with a callback that uses concrete type hinting for its parameter as object, therefore the mere fact of my plugin loading was breaking Joomla 4.2. Oops. As a result I figured that pausing the loading of plugins makes far more sense.

@roland-d I was at the beach, sorry. I needed a break after working my butt off yesterday. I will conjure a system plugin which is deliberately sabotaged for Joomla 4.2 and update testing instructions.

Now I understand.

Side not: Due some class loading conflicts with other extensions in the past, I stopped loading vendor in constructor and only when the respective listener function is called, in my plugins. Looks like I'm safe then here as well ?

@laoneo I had the opposite problem. Some dudes copied my vendor folder wholesale in their plugins, causing my plugins to fail if they tried to load vendor's autoloader in the event handlers. So I had to follow the opposite direction to yours. I understand it's a uniquely “me” problem (part of the burden of having my code being the missing reference for building Joomla extensions), I am just explaining the method to my madness ?

I have updated the PR and the testing instructions.

PS: I hope you appreciate my taste in music ?

@nikosdion

@roland-d I was at the beach, sorry. I needed a break after working my butt off yesterday. I will conjure a system plugin which is deliberately sabotaged for Joomla 4.2 and update testing instructions.

Of course, take your time on the beach, that is why I waited some hours from the morning :) I really appreciate your contributions. Joomla really benefits from your work.

So I have been following this further and I see a lot of discussion going on regarding the plugin handling and I really would like to see that moved to a separate PR so we can discuss/develop/test this further.

The part regarding the autoload_psr4.php deletion is tested and working fine and I want to merge that for the 4.2.1 RC release.

Could you do that please?

@roland-d Yup, will do. Let me fix the CLI application first and I'll come back to split this PR into two.

@roland-d I am making two PRs to replace this to avoid Git merge screw-ups.

The PR for the small change (removing autoload_psr4.php) is now live: #38525

I will post back when I create the second PR.

@nikosdion I understand, thank you very much.

Second PR: #38526

I am closing this PR as it's been superseded by the two split PRs.