@brianteeman I can't reproduce that here, starting with current staging (and the version patched from "3.9.22-rc" to "3.9.22-dev", because there seems to be another issue with version comparison when it contains "-rc", so the update to 3.10-alpha2 wasn't found, but that's another issue).

So I need more info:

• What was your starting point? Staging branch on a git clone, or a 3.9.21 installation?
• Server OS (I tested with Linux, don't have a server on Windows here)
• DB type and version: MySQL 8? Or pre 8? Or MariaDB?

@richard67 I think I have worked out the cause. Give me 10 minutes to confirm

I give you as many minutes as you need ;-)

OK so I managed to get further this time and I can see what is happening and what we need to fix.

The problem is with isSite and isAdmin which have been REMOVED in favour of isClient('site') and isClient('admin')

If you have a system plugin that uses those then it kills your site BEFORE the final stages of the update.

Normally you would do the update and then the site logs you out and you have to log back in and then enter your details again to complete the process.

The problem is that if you have a system plugin using either of those methods then the upgrade process dies at that point and there is no way to continue and you have a completely bricked web site

The isSite and isAdmin are VERY widely used and if we dont do something about this then its bye bye joomla.

You couldnt replicate it because you were testing with just core extensions.

It is a known issue https://docs.joomla.org/Potential_backward_compatibility_issues_in_Joomla_4#CMSApplication

It has been reported many many times https://github.com/joomla/joomla-cms/search?q=isSite&type=issues but I dont think anyone has reported before that it breaks upgrades and kills the sites. Just that a specific extension or template needed updating.

Unassigned me because it's not a database issue. But I agree it has to be solved.

Can it please be marked as a release blocker

The problem is that these functions specifically will not function as expected with the new console and api applications because they were largely used in the context of

if ($app->isAdmin()) { // stuff } else { // site stuff }

Or vice versa. I saw that pattern used lots (and even used it myself once upon a time)

We've ignored these reports since the change was made. Always blaming the developer for using a deprecated function and telling people just to a search and replace.

We can say its been deprecated for a few years as much as we want (I've said it myself) but when your site is dead you're not interested in assigning blame - you just want your site to work.

When it is in a system plugin then it kills the update and leaves the site in an unusable state that can only be recovered by starting again or manually running each sql script in the com_admin/ directory to even get to a state where the search and replace will be useful. That is assuming that you even know that you have to run those 72 sql scripts manually and know how to do it.

I suggested it before "why cant the old code be aliased to the new code" if the removal of the code really cant be reverted.

Finally - we can't even always blame the developer. Until recently the user profile plugin used this code and users were actively encourage to fork that plugin and use it as an example. I doubt that anyone who did that realised that the code needed to be changed. I know I didnt!

@wilsonge suggestion:
Create isSite()/isAdmin() only for SiteApplication and AdministratorApplication (and mark them depreciated)

In this way an old extensions will continue to work, and new extensions (which use Api/Cli Application) must use new method isClient() from CMSApplication

@brianteeman just a question not to blame anyone but what did the pre upgrade checker said for that extension? Has the Developer declared that extension to be 4.x compatible or not?

@wilsonge suggestion:
Create isSite()/isAdmin() only for SiteApplication and AdministratorApplication (and mark them depreciated)

In this way an old extensions will continue to work, and new extensions (which use Api/Cli Application) must use new method isClient() from CMSApplication

Hmm they are deprecated for a longer time already. I'm not sure whether another new method to deprecated them helps here.

The problem is that when that plugin is triggerd by an cli app and it uses isAdmin or isSite that would fail the cli app right as this would try to call an not defined method. Or i'm missing something?

No the pre-update check didnt say anything.

Just to remind you this is NOT about the offending code spitting out an error message or not working. That is easy to resolve. The problem is that this problem kills the update process leaving you with a partially updated site that is completely unusable and can only be recovered by completing deleting and reinstalling a backup.

No the pre-update check didnt say anything.

Just to remind you this is NOT about the offending code spitting out an error message or not working. That is easy to resolve. The problem is that this problem kills the update process leaving you with a partially updated site that is completely unusable and can only be recovered by completing deleting and reinstalling a backup.

Hmm can you send me the plugin in question? It should atleast say something.

I get the thing with the broken update. But that is how joomla works right now. We update the CMS with the same application that also runs other parts so all system events are triggered too. In an ideal world we would have an dedicated update app that would trigger dedicated plugin events only used in the updater.

I'm not sure how we could solve that as all other breaking changes introduced in 4.x could result into similiar issues when used in plugins.

Even an 'disable broken extensions' thing like Wordpress did would not help here i guess as it first need to fail in order to act and at this point the update is already broken.

And I'm not sure whether it is wise to make an conditional to all system plugin events that they should not run in the update process.

So i'm a bit out of ideas how to solve that other than in the extension itself.

sorry it did says something - the "no information available" message

This is not a case of a broken extension - we can live with that. This is a completely destroyed and unrecoverable web site

It is easy to test and see what happens. Just create a system plugin with code like this and do the update

	function onAfterRoute()
	{

		$app = JFactory::getApplication();

		if ( $app->isAdmin() )
		{
			return;
		}
	}

So i'm a bit out of ideas how to solve that other than in the extension itself.

Revert the removal of the code

if the extension is not ready for j4 you should not update your site.

sorry it did says something - the "no information available" message

Yes and that means that extension should be carefully checked upfront the Upgrade right?

This is not a case of a broken extension - we can live with that. This is a completely destroyed and unrecoverable web site

It is easy to test and see what happens. Just create a system plugin with code like this and do the update

	function onAfterRoute()
	{

		$app = JFactory::getApplication();

		if ( $app->isAdmin() )
		{
			return;
		}
	}

I know i just wanted the plugin to check why it did not show up in the pre Upgrade checker.

Btw you can break your site by calling any removed or broken code at that point even when you Upgrade from 3.9.21 to 3.9.22 (well there we have no sql stuff so you might be save here).

So i'm a bit out of ideas how to solve that other than in the extension itself.

Revert the removal of the code

I'm not sure whether that is a good solution too as mention by george. By that argument we have to revert all breaking changes out of 4.x as that might break the site on upgrade.

We might have to add just another more and bigger warning when system plugins are not compatible or no info is there?

Or we might disable extensions not compatible? That could break the site to but 'maybe' just the frontend and hopefully the backend still works?

We could also add just another warning and confirmation once you are upgrading when incompatible extensions are detected?

if the extension is not ready for j4 you should not update your site.

Of course not. That was NOT the case here. This was one of the million extensions that provided no update information

if the extension is not ready for j4 you should not update your site.

Of course not. That was NOT the case here. This was one of the million extensions that provided no update information

Any suggestion how to handle that? (other than reverting all breaking changes out of 4.x) Maybe one of my suggestions above can help with that? So special extensions like system plugins get a dedicated notice once found incompatibel or no info?

The problem is that when that plugin is triggerd by an cli app and it uses isAdmin or isSite that would fail the cli app right as this would try to call an not defined method.

That makes sense :)

Another idea, more fancy, but should keep us on safe side.
After compatibility check and before actually the upgrade:
The script store a map of enabled core/non-core extensions => turn off all non core extensions => make upgrade => restore disabled extensions.

With this we can be sure that 3d extension will not crash the upgrade (even if it say "it is compatible")

if the extension is not ready for j4 you should not update your site.

Of course not. That was NOT the case here. This was one of the million extensions that provided no update information

ok then this screen is missleading, if we hvae no information to a component then this component has to be marked as incompatible.

I tested one of my own components and ask my self which version the pre-update checker shows:

whats version 1.3.2 and why is it compatible?

after switching to https://update.joomla.org/core/test/310to4_list.xml it shows a big red block saying no update information available....

I would suggest to make a more explicit warning for major versions with known b/c breaks. If any extension is not marked as compatible don't let the user update without explicit confirmation.

The problem is that when that plugin is triggerd by an cli app and it uses isAdmin or isSite that would fail the cli app right as this would try to call an not defined method.

That makes sense :)

Another idea, more fancy, but should keep us on safe side.
After compatibility check and before actually the upgrade:
The script store a map of enabled core/non-core extensions => turn off all non core extensions => make upgrade => restore disabled extensions.

With this we can be sure that 3d extension will not crash the upgrade (even if it say "it is compatible")

sadly that will not work, because there are badly written plugins that thinks that the __construct function is a good place to execute code and joomla can't disabled this...

sadly that will not work, because there are badly written plugins that thinks that the __construct function...

hm, why? if extension is "disabled" it should not be even booted

Or can go even further, and turn off all extensions, leave only bare minimum, and then re-enable

sorry it did says something - the "no information available" message

Yes and that means that extension should be carefully checked upfront the Upgrade right?

If it does mean that then it needs to say it - which it doesn't

This is the output of my test site with 50% extensions providing no information including several very well known ones

updatereport.pdf

This is just being plain stubborn to prove a point that joomla extensions are not made by real developers. We know for a fact that the isSite/isAdmin change on a system plugin (at least) will break the update. We have no choice but to fix it. We can not bury our head in the sand and blame others. We've had reports of this since the beginning.

The problem is that when that plugin is triggerd by an cli app and it uses isAdmin or isSite that would fail the cli app right as this would try to call an not defined method.

That makes sense :)

That's a problem but it is not fatal. (ignoring the fact that a very small % will even use a cli app). If you get a failure with a message then you can react to it. This is a fatal error and would be the nail in the coffin for many joomla sites

sorry it did says something - the "no information available" message

Yes and that means that extension should be carefully checked upfront the Upgrade right?

If it does mean that then it needs to say it - which it doesn't

Can you suggest a better text for that message. I'm always happy to improve that upgrade checker.

This is the output of my test site with 50% extensions providing no information including several very well known ones

updatereport.pdf

This is just being plain stubborn to prove a point that joomla extensions are not made by real developers.

I dont think this has much todo with who did the plugin and whether there are "real developers" or not. But that set of extensions seems to be not marked compatible might just for the case that this developer did not tested it on 4.x

We know for a fact that the isSite/isAdmin change on a system plugin (at least) will break the update.

Yes and the same is true again for all b/c breaks we did.

We have no choice but to fix it.

I agree and I'm happy to discuss and improve the pre upgrade checker but i disagree to "just" revert that PR back as that would mean we have to revert any b/c break back.

We can not bury our head in the sand and blame others. We've had reports of this since the beginning.

No I do not I have suggested workarounds that works generally not only for isAdmin/isSite.

Additionally to my last comment.

Main goal is to make a core upgrade without crash.
Literally execute update #_extension set state = 0 where <non core>, and load page with upgrade process (it important for unload disabled extensions).

After upgrade succeed set state = 1 back. It may lead to crash, but at this point we sure that the core is updated and can handle fatal errors.

We know for a fact that the isSite/isAdmin change on a system plugin (at least) will break the update.

Yes and the same is true again for all b/c breaks we did.

No it is not. That is the point.

It is perfectly acceptable for an extension not to work because of b/c breaks. That is expected (at least by me). I was actually preparing a tutorial video to show people what to do after an update when an extension doesn't work.

What is not acceptable is for the update to fail and leave you with a site in a state that can only be recovered by completing erasing it and reinstalling a backup.

Main goal is to make a core upgrade without crash.
Literally execute update #_extension set state = 0 where , and load page with upgrade process (it important for unload disabled extensions).

What happens with extensions that are expected to execute code before or after the update. similiar to the "backup before update" plugin from akeeba?

We know for a fact that the isSite/isAdmin change on a system plugin (at least) will break the update.

Yes and the same is true again for all b/c breaks we did.

No it is not. That is the point.

well the same happens when you have a plugin that uses JMail::sendAdminMail (https://docs.joomla.org/Potential_backward_compatibility_issues_in_Joomla_4#Mail) or use JString. (https://docs.joomla.org/Potential_backward_compatibility_issues_in_Joomla_4#String)

isAdmin and isSite is more common for sure.

What is not acceptable is for the update to fail and leave you with a site in a state that can only be recovered by completing erasing it and reinstalling a backup.

I agree that is the reason the pre upgrade checker lists all extensions to be checked upfront within 3.10 not after the upgrade.
What about the suggestions I did above for that special plugins that can acctually break the upgrade so we catch that issues before the update happens?

What happens with extensions that are expected to execute code before or after the update

It depend "when" we disable them, and when the Upgrade event will be triggered.
Could be like this:

• check extensions compatibility
• trigger onBeforeUpgrade
• execute #_extension set state = 0
• reload page and run upgrade
• execute #_extension set state = 1
• reload the page to boot disabled extensions
• trigger onAfterUpgrade

What happens with extensions that are expected to execute code before or after the update

It depend "when" we disable them, and when the Upgrade event will be triggered.
Could be like this:

• check extensions compatibility
• trigger onBeforeUpgrade
• execute #_extension set state = 0
• reload page and run upgrade
• execute #_extension set state = 1
• reload the page to boot disabled extensions
• trigger onAfterUpgrade

Ok but this has to be implemented in 3.10 and by that change we break existing plugins using the old way right? That would be a B/C break or I'm missing something? That could be a workaround we could implement into 4.x for the next updates. And in this specific case of a system plugin calling an undefined method at onAfterRoute would still break the site and the backend.

I agree that is the reason the pre upgrade checker lists all extensions to be checked upfront within 3.10 not after the upgrade.

Did you look at the list I shared - no way anyone (even a hard core dev like yourself) is going to check that

/me off to debug a different update issue with sql

by that change we break existing plugins using the old way right? That would be a B/C break or I'm missing something?

we do not break anything we just turn them off, at 1 step before actually replacing files and run sql update.
upd: or maybe I missing something

And in this specific case of a system plugin calling an undefined method at onAfterRoute would still break the site and the backend.

While upgrade: the plugin is disabled, so no break.
After upgrade: yes it will break, but at this point the core upgrade is finished, and the exception handler will show what is going on, so User can guess where to look.

Did you look at the list I shared - no way anyone (even a hard core dev like yourself) is going to check that

Well i would check them as i know the consequences. But yes thats the reason i suggested the message for the system plugins that could break the upgrade any feedback on that idea is welcome.

we do not break anything we just turn them off, at 1 step before actually replacing files and run sql update.
upd: or maybe I missing something

Well theoretical some extension could listen on a specific update step to execute some code like the backup on update plugin or like set the site in some kind of update mode or whatever bussines logic makes sense at this point. That usecases would be broken and to me this would be a B/C break that can not go into 3.x but has to be in 4.x where we also have to solve the thing that we finish the update with a different codebase as we started from so for the first version we add such thing we have to take care that this update to enable the extensions again is not breaking ;)

While upgrade: the plugin is disabled, so no break.
After upgrade: yes it will break, but at this point the core upgrade is finished, and the exception handler will show what is going on, so User can guess where to look.

Agree but i dont think someone should update to 4.x when there are known issues with such central plugins.

You would check all the extensions in that list? Really?

The pre-update check does not report what type of plugin it is

Agree but i dont think someone should update to 4.x when there are known issues with such central plugins.

They do not know there are such issues

You would check all the extensions in that list? Really?

Well I personally would not install that many extensions in the first place but yes i would check why they fail or atleast disable them manually upfront. That would also work even without B/C.

We might mention that workaround disable the extensions listed here in the pre upgrade checker can you suggest a good wording for that?

The pre-update check does not report what type of plugin it is

That is one of the simplest thing to add when that helps but a dedicated message for system plugins too.

They do not know there are such issues

Well there are 40 extensions that joomla can not check / say whether it is compatible, that is the issue right? I would asume that some of them fail.

theoretical some extension could listen on a specific update step to execute some code like the backup on update plugin or like set the site in some kind of update mode or whatever bussines logic makes sense at this point

that why I have separate it to a different steps ;)

• trigger onBeforeUpgrade - the extension doing its dirty job, eg: the backup on update set the site in some kind of update mode or whatever
• execute #_extension set state = 0
• reload page and run upgrade - write new files, update DB, at this point no one can Interrupt upgrade except hidden bug in a core
• execute #_extension set state = 1 - here we should skip "known incompatible"
• reload the page to boot disabled extensions
• trigger onAfterUpgrade - the extension doing its dirty job, eg: the backup after update or set the site mode

btw, do we have a loot of events while core update?

I'm giving up - I will not participate in any discussion where you deliberately misquote me. And you clearly did not actually read the extension list I posted.

that why I have separate it to a different steps ;)

Yes but that new plugin event has to be implemented by the plugin first right? as the old way using onAfterRoute is not working anymore. -> B/C break ;)

I'm giving up - I will not participate in any discussion where you deliberately misquote me.

I'm sorry for that. It was not my intention to miss quote you where did i did that? So we can clear things up.

And you clearly did not actually read the extension list I posted.

I checked the list you posted but what is the point? That there are old fof stuff listed? That dpcalendar is listed? I seem to miss something in that context?

btw, do we have a loot of events while core update?

well that list here for sure: https://docs.joomla.org/Plugin/Events/System as a full page is loaded a few times probertly a few more too.

well that list here for sure: https://docs.joomla.org/Plugin/Events/System

I meant "update" specific,
well I just found onJoomlaAfterUpdate, and nothing for "Before"

Yes but that new plugin event has to be implemented by the plugin first right?

I did not meant new events ;)
I thought there already exists Before and After update, but seems only After exist, that make thing a bit complicated ?

Okay, may need to split &task=update.install to 3 (or more) tasks, and modify &task=update.finalise:
1 &task=update.install - at this point we doing nothing, just redirect to next task, this allows to extensions hookup as usual.
2 &task=update.step1 - here the script store what extensions was enabled, and disable all of them, then redirect to next task
3 &task=update.step2 - here is where the files update happens, and redirect to next task
4 &task=update.step3 - this is existing update.finalise, it execute update script, update SQL queries, and redirect to next task
5 &task=update.step4 - restore previously disabled extensions, redirect to next task
6 &task=update.done - here run onJoomlaAfterUpdate and show "Hello world!" message

Important that between step1 ... stepX should not be allowed to run any extension.

There may be more or less steps (example can split to 2 update script and update SQL if need), or they may be ordered differently.

In general it just an idea.

Please just do what I asked. Create a system plugin with the removed code and see what happens. It is NOT that extensions may not work on an upgraded site. Everyone accepts that. The problem is that this "pointless" code change results in a failed update. It leaves the site in a state that can neither be fixed or reverted. the only option is to completely wipe the site.

Reopening issue as not solved.

@brianteeman would be nice to have more solid fix that prevent :

It leaves the site in a state that can neither be fixed or reverted. the only option is to completely wipe the site.

It happen because how Update process works:
&task=update.install - replace new files, then new request to
&task=update.finalise - here should be executed update script and SQL update, but it never happen because buggy extension is booted and doing a dirty stuff onAfterRoute and before actual update.finalise executed. That break update process and leaves the site broken, because outdated database schema.

It may happen with any other method used by extension that was removed in j4, but existed previously.

Yeah we can just restore isSite/isAdmin, but this bug may come up in another form, later.

well that list here for sure: https://docs.joomla.org/Plugin/Events/System

I meant "update" specific,
well I just found onJoomlaAfterUpdate, and nothing for "Before"

Yes but that new plugin event has to be implemented by the plugin first right?

I did not meant new events ;)
I thought there already exists Before and After update, but seems only After exist, that make thing a bit complicated ?

Okay, may need to split &task=update.install to 3 (or more) tasks, and modify &task=update.finalise:
1 &task=update.install - at this point we doing nothing, just redirect to next task, this allows to extensions hookup as usual.
2 &task=update.step1 - here the script store what extensions was enabled, and disable all of them, then redirect to next task
3 &task=update.step2 - here is where the files update happens, and redirect to next task
4 &task=update.step3 - this is existing update.finalise, it execute update script, update SQL queries, and redirect to next task
5 &task=update.step4 - restore previously disabled extensions, redirect to next task
6 &task=update.done - here run onJoomlaAfterUpdate and show "Hello world!" message

Important that between step1 ... stepX should not be allowed to run any extension.

There may be more or less steps (example can split to 2 update script and update SQL if need), or they may be ordered differently.

In general it just an idea.

Good idea for sure. Do you think you can create a PR for that? Some of the update logic is a mix of JS and PHP and tbh I'm not that familiar with JS ;-)

As this would be a B/C break i think this has to go aganinst 4.0.

I can try, only it will be not fast :)

As this would be a B/C break i think this has to go aganinst 4.0

I think we need it in 3.10, to prevent crash while transition 3.10 => 4.0
It could be done with keeping b/c, all we need is to keep first task name &task=update.install and last task name &task=update.finalise (instead of &task=update.done as I wrote) and trigger onJoomlaAfterUpdate as it currently.

I do not see other b/c issues.

I can try, only it will be not fast :)

No problem. Feel free to contact me when you have a question so we can move that forward together. ?

As this would be a B/C break i think this has to go aganinst 4.0

I think we need it in 3.10, to prevent crash while transition 3.10 => 4.0
It could be done with keeping b/c, all we need is to keep first task name &task=update.install and last task name &task=update.finalise (instead of &task=update.done as I wrote) and trigger onJoomlaAfterUpdate as it currently.

I do not see other b/c issues.

Great idea! Yes that could work in a B/C way.

At that step we can also add a dedicated onJoomlaBeforeUpdate so noone is required to do it that hacky way anymore. ;-)

I dunno how @nikosdion is doing backups but I'm pretty sure disabling all system plugins is going to cause issues with things like the backup before update plugins (and other such extensions too).

How about we just catch the exceptions in system plugins and continue? Even better would be from the stacktrace trying to disabling the plugin that caused the issue (Like what wordpress announced at some point)

How about just accepting that the removal of this code was a mistake

I dunno how @nikosdion is doing backups but I'm pretty sure disabling all system plugins is going to cause issues with things like the backup before update plugins (and other such extensions too).

Yes thats the reason we want to keep the old processes to work. So they are still fired and can run the thing they need but they can not break the upgrade.

How about we just catch the exceptions in system plugins and continue? Even better would be from the stacktrace trying to disabling the plugin that caused the issue (Like what wordpress announced at some point)

Yes i thourgt about that too the problem is at this point the upgrade is already broken right? As even for wp you need a site reload to restart without that plugin.

How about just accepting that the removal of this code was a mistake

Its not about that specific code it is an issue that any kind of PHP error caused for whatever reason at that point break the upgrade.

Really I don't understand the problem, a proper communication that one of your extensions is not j4 prove and you should not updated until all installed extensions are compatible.

Trying to disable all system plugins doesn't mean that fix everything because we also have auth plugins that maybe have to be active. And I'm pretty sure there are other code locations which could lead to an update problem if a 3rd party extension is not prepared for j4.

Disabling all system plugins is extremely ill advised. Anyone who understands how Joomla works knows how likely it is for the update to fail and understands what disabling all system plugins does to the site. Not to mention how extremely unlikely it is even for a hardcore developer to bounce back from that without restoring from a backup (losing all information between the last backup and Joomla pooping its pants on update). A "solution" which effectively causes the software to malfunction to the point of total unusability every time an update fails is just plain bad and merits no second thought.

Regarding my backup on update plugin. I have a system plugin which sees the page request to start the update after the download is complete. At this point it redirects to Akeeba Backup in a way that starts a backup. Upon backup completion the original redirection is effected and honored.

I will just say this. Updates are extremely important and very hard. It is not a subject that should be left for amateur hour. If you can't handle these requirements please let a responsible adult help you. Things like that make or break a product.

disable all system plugins

who said "all"? I suggested "non core" or leave "bare minimum", and only at time while executed "update script" and "update SQL"

you should not updated until all installed extensions are compatible

Yeah, that also true, or they should be disabled by Admin manually.
That maybe more smart.

How about we just catch the exceptions in system plugins and continue?

offtopic: there was a script called something like "make this sh*t work", it wraps the code in try/catch, and in case of an error it removes buggy line of code then execute again, it continue until the code will be executed without errors :)

Disabling all system plugins is extremely ill advised. Anyone who understands how Joomla works knows how likely it is for the update to fail and understands what disabling all system plugins does to the site. Not to mention how extremely unlikely it is even for a hardcore developer to bounce back from that without restoring from a backup (losing all information between the last backup and Joomla pooping its pants on update). A "solution" which effectively causes the software to malfunction to the point of total unusability every time an update fails is just plain bad and merits no second thought.

Ok got your point any suggestions to over come that mention problem?

The siggestion was to disable the plugin just for the few minutes that the update process is run after that they should have been enabled again and possibly break the site but not the upgrade.

Regarding my backup on update plugin. I have a system plugin which sees the page request to start the update after the download is complete. At this point it redirects to Akeeba Backup in a way that starts a backup. Upon backup completion the original redirection is effected and honored.

Yes it is the only plugin that i'm aware of doing things there.

I will just say this. Updates are extremely important and very hard. It is not a subject that should be left for amateur hour. If you can't handle these requirements please let a responsible adult help you. Things like that make or break a product.

Ok do you have any suggestions to move that into a good state? You worte the update system so i'm happy for your input here.

disable all system plugins

who said "all"? I suggested "non core" or leave "bare minimum", and only at time while executed "update script" and "update SQL"

Yes we could also just disable all plugins not maked as compatible but i think such a thing needs to be thinked throught.

you should not updated until all installed extensions are compatible

Yeah, that also true, or they should be disabled by Admin manually.
That maybe more smart.

Yes that is the other thing proposed here. Issue a message and suggest to disable the extensions possibe incompatible extensions for the time of the upgrade manually.

I dunno how @nikosdion is doing backups but I'm pretty sure disabling all system plugins is going to cause issues with things like the backup before update plugins (and other such extensions too).

How about we just catch the exceptions in system plugins and continue? Even better would be from the stacktrace trying to disabling the plugin that caused the issue (Like what wordpress announced at some point)

yep.. just FYI i have try update for testing purposes on a clone website client full of sys plugs with isSite() with akee**** extension and more and more sh*t and yes @brianteeman was totally right . website is now a brick ..
also FIY to test how unrecoverable is that thing ive tryed to point the original/clean website to the database where the update failed .. and it does not work. so the problem bricked also the DB in some way (please do not ask me how)

Really I don't understand the problem, a proper communication that one of your extensions is not j4 prove and you should not updated until all installed extensions are compatible.

and honestly as "developer" and "administrator" i like what @HLeithner said. extension not ready for j4 (or without info about is ready or not) you do cannot update or radio button "go ahead i know what i am doin - im ready to destroy it".

@ceciogit I am offended by your comparison of my software to feces and your false assertion about how it works. I expect an apology at the very least.

Akeeba extensions DO NOT use isSite/isAdmin. Detection of the active application is part of FOF 3 since 2015 and I have not been using isSite/isAdmin since February 2019. As you can see the code is structured in a way that will work with all Joomla versions, before and after the isAdmin deprecation and the introduction of isClient. This was part of my extensive work at Joomla 4 compatibility nearly two years ago. Even before that, the way the code is structured would fail gracefully, without bringing your site down. I was already aware of similar issues in other deprecated APIs and practices defensive coding. You don't have to believe my words. I prefer letting my code do the talking. You can see for yourself the unforgeable history of my code on GitHub.

Furthermore, I have spent an obscene amount of time making sure that ALL of my extensions are Joomla 4 compatible, even though Joomla 4 still changes on a weekly, if not more frequent, basis despite it being tagged a beta. I was the first ever Joomla extensions developer to support Joomla 4, long before the JED even allowed us to add a J4 tag to our extensions. Let alone the fact that I was the person who kickstarted the Joomla 4 development five years ago in Prague...

Having my Akeeba extensions installed on your site WILL NOT prevent you from upgrading to Joomla 4 or cause any problems after upgrading as long as you have the latest version installed. Now, if you have leftover plugins from 3-8 years ago you never removed and which were not updated (because they got removed) that's on you.

I would also like to point out that I'm not just offering a philosophical view here to defend the software I have written without any evidence. I have already put in real work to upgrade a real world site from J3 to J4.I took a backup of our site, akeeba.com, and restored it on my dev server. That's a rather complicated Joomla 3.9 site build. I went from 3.9 to 3.10 alpha to 4.0 beta.

I am using the 4.0 beta dev site the past ten days to convert our template to Joomla 4. I've already tried the minimum effort method which keeps deprecated API calls to understand the typical experience of a busy site integrator. I am currently putting in the work to write a template from scratch, using Bootstrap 4, to better understand the experience of a more advanced site integrator who is undertaking a larger scale migration project. The idea is that my clients will go through that process so I need to be sure I can help them when they seek assistance with regards to my software. I can happily report that I have only experienced minor issues with our software which I have already fixed in the releases published last week.

As for the actual upgrade process I have a few notes. Like y'all said, it is a mess but not because of anything I am doing in Akeeba software (and believe me when I say that I run all of it on our site, including those extensions marked as FOR INTERNAL USE ONLY in our GitHub repos).

Upgrading from 3.9 to 3.10 beta requires first running a SQL command which is documented in an obscure Joomla Community blog post and which was woefully wrong – I was the one who notified @wilsonge about it so he could fix it. You're welcome.

Updating 3.10 to 4.0 consistently failed -- I have some ideas why but I didn't have the time or the inclination to properly debug it. Even removing all Thord party extensions, including my own, and reverting to the core htaccess code didn't help. Instead I applied the update from the command line. I modified the post-update script in com_admin to run under CLI so I could complete the update without having to go through com_joomlaupdate which was crashing on me. Success. Kinda.

I have exactly two (2) third party, non-Akeeba extensions on the site: JCE and EasyUserLogin. The former works perfectly. The latter, written by former core contributor nonetheless, was causing the site to crash because it's using the obsolete CMSApplication methods isSite/isAdmin. The only way to find that out was to set error reporting maximum and log PHP errors to a file. Once I found the culprit I disabled the plugin from the database and the site works perfectly.

If anything, Akeeba Backup saved me a ton of time because I was taking backups every step of the way. 3.9 to 3.10: backup. 3.10 to 4. Backup. Those two backups got rolled back enough times to make you dizzy while I was battling with the confusing and incomplete migration instructions. So there's that.

i really cannot undesrtand, blame me if you want, in this case i really don't mind, but after 1y of @deprecated notice published on docs etc...
and with a pre-update check that shows you to take attention about not ready for j4 3dp......

really i'm out you cannot fix stupid

@alikon because you are thinking as a developer and not as a user/site owner AND because even if you were a developer you would not have had any deprecation notices in your logs about this.

Thanks for calling me stupid. Been called much much worse.

@alikon look at the example from @nikosdion He has isSite in his code but he wrote it so that its not a problem. His other extension also had isSite in the code but it was a problem.

If both of those extensions did not yet have a j4 compatible flag are you really expecting a site owner to go through both those components and understand which occurrences they have to change and which ones they can leave alone.

I wouldn't have a clue

You are also assuming that everyone is using extensions from mass market developers. Very many people have code written for them so none of that code will have the J4 compatible flag and every line of that will need to be checked.

@alikon Finally even you were submitting code like this to core #23335 despite it being flagged as deprecated #22683

#23335 was from the past Dec 26, 2018 some thing like 2 y ago isn't it?
but nevermind

i'm sorry but i cannot follow you, all your points are specious

this discussion is going nowhere imho

currently we have :

• followed SEM
• setting @Deprecation from more than 1y already in 3,x
• writing docs https://docs.joomla.org/Potential_backward_compatibility_issues_in_Joomla_4
• having a pre-update check

what do you want more ??

for a magic wand we are still working ?

It is going nowhere because you are not thinking like a user. My point of mentioning that PR was that it was created AFTER the deprecation. Even the author of the PR that removed the code in J4 uses it extensively in the very latest release of their software.

IF a user would see those docs AND
IF the deprecation was in the logs AND
IF the pre-update check said NOT compatible THEN
I agree with you

Clearly developers haven't seen those docs so we cant expexct users to see AND understand them so we fail at step 1
The deprecation isnt in the deprecation logs see #30926 so we fail at step 2

But fine keep sticking your head in the sand and ignoring the problem. If two very experienced users failed to update with the component then its a real problem. If you want to be responsible for the end of joomla fine but I dont.

hi @nikosdion , far from me associate akeeba bkp to feces. are you serious? akeeba is an extension i love and use from years.
finally this is the prove my english sucks. lol. moreover because i had to be crazy answering to you telling your extension is sh*tty.. gosh..

i was refering to akeeba for the "backup b4 update" AND other sh*tty extensions that block the update. akeeba was not mean being part of feces plugins.

i am really sorry for misunderstanding.

@ceciogit Thank you for your reply. I understand that it was just a misunderstanding because of the language barrier. Apologies for my reaction. I misunderstood you. Big hug.

@alikon Both you and @brianteeman are partly right and partly wrong. From a developer's perspective I understand why isSite and isAdmin had to be removed: we now have at least for different applications (site, admin, CLI and API) in Joomla, these two methods don't make sense in the CLI and API applications.

However, Brian is right in that the deprecation in SiteApplication and AdministratorApplication did NOT come with a deprecation notice in the log which is what Joomla recommends developers consult to detect deprecated features in their code! Moreover, using isSite/isAdmin is so ingrained to Joomla developers that it's hard to kick the habit. The proof is that the person who committed the obsolescence of these methods still uses them in his own production code even today. Therefore we need a programmatic solution to a. not break sites and b. gently inform developers they should start replacing these obsolete calls.

There is an elegant code solution. Create a Trait which implements isSite/isAdmin as shims to isClient and uses trigger_error to issue an E_USER_WARNING and logs the use to the deprecated log. The message in both latter cases should be "Using isSite() is deprecated and will remove. Use isClient('site') instead." You get the idea. Use this trait ONLY in SiteApplication and AdministratorApplication and mark it as @deprecated 5.0. This solution will prevent the bulk of migrations fail and gives adequate feedback to developers to fix their code. This should have been done in 3.8 but, hey, we can't travel back in time.

Yes, these system plugins will still break the CLI and API applications. Big effin' deal. The target audience of these applications is vert strongly correlated with the target audience of the deprecation notice. If you want to be overly pedantic create another Trait for these two applications which throws a RuntimeException when either method is called. This will provide even stronger feedback to the developers who will test their code with the new, non-HTML, non-web applications of Joomla 4.

The bottom line is that you do not need to shoot your feet by disabling system plugins on update (that's an insane non-solution which cause deeper and bigger problems). You don't need to rage-quit. You don't need to bicker forever over spilt milk. There are always elegant, obvious (or not-so-obvious), programmatic solutions.

Use this trait ONLY in SiteApplication and AdministratorApplication and mark it as deprecated

That what was suggested at beginning #30913 (comment)

do not need to shoot your feet by disabling system plugins on update

Why a plugins need to be enable at time when executed "update script" and "update SQL"? There nothing to do for them.
Joomla update have isolated script for "unpack" the files, but not for further critical steps. That why it so easy to explode.

However I think we already come to conclusion, that @HLeithner idea "to block the update if there an extension that cannot confirm compatibility" will be a good solution.

However I think we already come to conclusion, that @HLeithner idea "to block the update if there an extension that cannot confirm compatibility" will be a good solution.

Hmm, correct me if I'm wrong, but as far as I remember the 4.0 compatibility information comes from the update server's XML file. If that is right, only extensions using update servers can confirm compatibility. Extensions which have to be installed or updated "manually" with zip upload will always be listed as with unknown compatibility status. Shall these block updating the core?

If that is right, only extensions using update servers can confirm compatibility.

hm, If it true, then we have one more issue that need to fix somehow :)
Because "self made" extensions will not have "update server".

Every extension XML have version="3.xx" thing:
<extension type="component" version="3.xx" method="upgrade">
Maybe we can use it somehow for local extensions, to allow something like version="3.10,4.0".
But I am not sure here.

@Fedik Fair points and they do have reasonable answers ?

Why a plugins need to be enable at time when executed "update script" and "update SQL"? There nothing to do for them.

Top reason off the top of my head: you are using a MySQL cluster which necessitates installing a custom database driver, only possible with a system plugin. Disabling the plugin would update the (meant to be read-only) slave but not the (meant to be write-only) master.

The other reason is that system plugins are loaded very early in the application boot, before you can run custom code. So in the failure mode that we are discussing here you'd have to disable these plugins in the database. However, this can only be done BEFORE you start the extraction. This means that an extraction problem or a post-install script failure would end up with your site's system plugins being permanently disabled. Let's say that magically this wasn't an issue. Where you'd store the IDs of the system plugins previously enabled? It can't be the session because there's no guarantee it will still exist at the end of the update (especially when you're moving to a new major version). You'd have to store them in the database and modify the post-installation script to use that information. This makes recovery from a failure possible but VERY HARD for the target audience. You could try to solve it by having the application always look for that information in the DB and enable system plugin the next time it boots up, before it looks for system plugins. However, this now opens an exploitable security hole. I can think of at least three fun ways I can hack your site with file-only access and you'd be none the wiser – my evil plugin would never show up enabled and you'd see no modified files.

Another solution which kinda helps is having the post-installation script NOT implement the application interface. This is problematic on many fronts, especially maintainability. This is EXACTLY what I did when I was upgrading my dev site. The worst part is that it doesn't solve the post-update problem which is also EXACTLY the problem I ran into. Let me tell you why this is bad. Joomla 4 implements AJAX login on the backend. Logging into my restored site failed consistently. I could not figure out why until I edited configuration.php to set error reporting to maximum AND enabled logging of PHP errors to a file in my php.ini. Then and only then could I see the exception in my PHP error log which pointed the finger to a 3PD plugin. There was sod all to help me on the browser. Just that the login failed. As a user my experience was that J4 was broken. As a developer I was left screaming "who the fuck thought this is a good idea?!".

So, why go through all that pain when a Trait would do? IMHO the simplest solution is always the best. Don't add a thick layer of obscurity which relies on institutional knowledge which will disappear when those of us participating in this discussion stop contributing for any reason. If the past 15 years taught us anything is that every time we implemented a convoluted solution it resulted in a clusterfsck two to four years down the line when institutional knowledge was lost due to volunteer turnover and some well-meaning volunteer committed something which made the whole thing blow up.

Regarding extensions without an update server, they should never be update blockers. They should simply be marked as unknown status which is the current state. Anything else will be perceived by users as a bug which makes updates impossible. They'll try to McGuyver the update with catastrophic results.

PS: Maybe y'all should try upgrading REAL WORLD sites to Joomla 4 to see what REAL WORLD users will experience. Y'all are trapped in testing clean installs. Yeah, sure, clean installs migrate (relatively) easily. Big deal. The devil is in the details and you need to take into account as many details as possible. Your users don't care if the problem comes from Joomla core or a 3PD extensions. They perceive Joomla 4 to be broken.

Dammit, people! I tried to tell you in person in January and almost nobody would listen. Users don't give a crap about whether you're using MVC, if you have middleware and shit like that. THEY USE WORDPRESS FOR CRYING OUT LOUD! Its code is a hot mess. They still use it because they don't have to think hard about updates the vast majority of time. When Joomla offers a subpar user experience they WILL go to a different CMS, code quality be damned. They just want to get shit done instead of battling with the software. It feels like I am screaming into the void again...

Okay, that makes sense. With some part I agree and some not, but this not for this topic.

Regarding extensions without an update server, they should never be update blockers.

While we talking @brianteeman made a concept #30930 :)

My concept would still be much much better with the code @nikosdion proposes which I have been trying to say in my non-developer language for almost a year #24642 (comment)

added some missing deprecations Log #30938

The trait suggested by @nikosdion would seem to be a good idea and complimentary to #30930 - the downside is that it only catches a couple of the possible problems that may arise. I just ran a test upgrade and found some code using JFactory::getAcl() - there will be other offending calls from plugins so we are unlikely to catch them all if users try to upgrade a site with a plugin that isn't compatible.

I hesitate to wade in to this one with a different (or complimentary) approach - but the problem Brian encountered was specifically during com_joomlaupdate and task=update.finalise to avoid bricking a site during the upgrade so we only really need to deal with this situation. An upgraded site with a broken extension is more straightforward to resolve so the goal should be to get to successfully upgraded site even if an extension is still broken??

Using try/catch in the triggerEvent method of EventAware seems to get around this (from my limited testing)

`
try
{
$result = $dispatcher->dispatch($eventName, $event);
}
catch ( \Throwable $throwable)
{
$this->getLogger()->error(sprintf('Dispatcher not able to trigger event %s ', $eventName));

		return [];
	}

`
But leaving this code modification in after the upgrade is completed is almost certainly not a good idea. Doing something specific to the context of "update.finalise" would be really ugly code. Not sure of how we could make the code 'specific' to the context since the onAfterInitialise is called before the component code.

I would personaly would not like to introduce specific code for the update process. When there is a plugin that breaks than it should break.

As mention also in the other issue i'm working on a proposal / RFC for the pre Upgrade checker that issues a warning / error once there a system plugins with unknown compatibliy enabled and requests another dedicated SU confirmation that this site should possible be bricked. It would be an additional check to the pre upgrade checker that also gets improved with the strings that brian proposed in the other issue. So we have a much clearer message to our users.

But please give me a bit more time to come up with that PR.

And all that without breaking legimitiate use cases like the system plugin that loads the database handler. Or trying to catch an issue or just partly patch it and break other parts of the application.

An upgraded site with a broken extension is more straightforward to resolve so the goal should be to get to successfully upgraded site even if an extension is still broken??

Yes

Yes i agree on that too. :-)

The main thing is we have to make sure that the update SQL scripts are run, because a later db fix will only fix structure (if we could come to that point that the fix would be possible), but many problems with broken backend after broken update come not only from structure but also from data, e.g. menu items missing in backend in the admin menu (menu type "main").

Or we make sure that the db fixer can be used and that it can detect a broken update by the schema version in database and offers to finalize the broken update.

The main thing is we have to make sure that the update SQL scripts are run, because a later db fix will only fix structure (if we could come to that point that the fix would be possible), but many problems with broken backend after broken update come not only from structure but also from data, e.g. menu items missing in backend in the admin menu (menu type "main").

Or we make sure that the db fixer can be used and that it can detect a broken update by the schema version in database and offers to finalize the broken update.

Yes that is the reason we want that only confirmed compatible extensions are enabled to run at that point of the upgrade so any issue they might produce does not brick the updgrade.

The impact of such error with a large upgrade like 3.10 to 4.x is much higher than on smaler updates for sure. But IIRC the finalise step is also not only about the database stuff.

In the end we have to make sure that all steps for the upgrade run correctly and that any plugin that could break that upgrade is checked upfront.

Yes, but incompatible plugins are not the only problem. If you just break the database connection at the wrong point of time during the update, when running the first few SQL update scripts, then you end with a brick, too. Not sure if this can be easily fixed.

@richard67 You can. I've been doing that in Akeeba Backup for 6+ years. If the DB query fails you check if the connection still pings. If not, reconnect and run the query again.

The biggest problem Joomla has during updates is that its schema update mechanism is NOT idempotent. You have a bunch of SQL files with an arbitrary number of queries which must succeed all at once or the whole update is hosed to oblivion. This does happen in real world usage.

At one point it was suggested that we use Doctrine exactly for this reason. The problem is that Doctrine is a rather thick, very opinionated abstraction layer which doesn't let us apply the kind of fine grained optimisations we need per database server type.

The solution I came up with several years ago was my own, mostly idempotent, schema installer and updater. An XML file wraps blocks of SQL queries in conditions. The idea is that before running the SQL query/-ies the condition is true, afterwards it is false.

As soon as I started using it in my software the number of SQL issues on update dropped to zero and I was able to add an automatic schema fix feature regardless of the state the schema was in. There was a radical drop in support requests with regards to database issues.

I did try to get that, or something similar (and improved -- I now know the limitations of my design) to Joomla 4 but... let's say politics won and leave it at that.

Please find here a RFC regarding the issue discussed here: #31042

@richard67 You can.

If the "you" means "one", i.e. anyone: Ok.

But if it means me personally: Not sure if I can. I have limited time and currently also limited energy to work on that. At least I can't do it alone. I might help with the one or other thing though.

Closing as there is now a PR from @GeraintEdwards at: #31200. Thanks!