J4 Issue ?
avatar richard67
richard67
9 Apr 2020

Steps to reproduce the issue

Use a MySQL or MariaDB database with the "MySQLi" driver, i.e. not the "MySQL (PDO)" driver.

Install a current 3.10 nightly or 3.10-dev branch.

At the end of the installation, chose the additonal step for installing languages and install French and Persian.

When languages have been installed, chose to enable multilanguage and select to install localized content.

Then finish the installation and login to backend.

Go to the Joomla Update component and update to last 4.0 nightly build, either using the Live Update with following custom URL https://update.joomla.org/core/nightlies/next_major_list.xml or the Upload & Update with the previously downloaded update package from here https://developer.joomla.org/nightlies/Joomla_4.0.0-beta1-dev-Development-Update_Package.zip.

Start the update and enter credentials whenever asked for it.

Expected result

Update successful.

Actual result

The update starts to unpack the update package.

Then immediately after unpacking of the package:

j4-update-multilingual-error-500

In administrator/logs/joomla_update.php:

2020-04-09T10:43:58+00:00       INFO 192.168.98.1       update  Update started by user Super User (672). Old version is 3.10.0-dev.
2020-04-09T10:43:58+00:00       INFO 192.168.98.1       update  Downloading update file from https://developer.joomla.org/nightlies/Joomla_4.0.0-beta1-dev-Developmen
t-Update_Package.zip.
2020-04-09T10:44:06+00:00       INFO 192.168.98.1       update  File Joomla_4.0.0-beta1-dev-Development-Update_Package.zip downloaded.
2020-04-09T10:44:06+00:00       INFO 192.168.98.1       update  Starting installation of new version.

I.e. it fails before the first SQL statement of any update SQL script is executed or with the very first statement. Maybe a problem with connecting?

The log of the database server shows nothing,

System information (as much as possible)

PHP 7.2, MySQL 5.7, Apache 2.4 on Ubuntu.

Additional comments

I happens too when updating a staging to 4.0 nightly (for which Upload&Update has to be used because the Live Update will not find an update for 3.9 to 4). So it seems not to be a 3.10-dev issue which has been fixed already in staging.

It does not fail when no localized content is installed, i.e. only the languages are installed but no content.

It does not fail when using the "MySQL (PDO)" driver.

It does not fail when using the "PostgreSQL (PDO)" driver with a PostgreSQL database.

I don't know when it has stopped working. I have tested many updates here in recent weeks, but not with multilingual sites having language-specific content. Thanks to @infograf768 who first discovered this issue today.

avatar richard67 richard67 - open - 9 Apr 2020
avatar joomla-cms-bot joomla-cms-bot - change - 9 Apr 2020
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - labeled - 9 Apr 2020
avatar infograf768 infograf768 - change - 9 Apr 2020
Labels Added: ?
avatar infograf768 infograf768 - labeled - 9 Apr 2020
avatar infograf768
infograf768 - comment - 9 Apr 2020

Confirmed. Set to Beta blocker.

avatar richard67
richard67 - comment - 9 Apr 2020

@infograf768 Please confirm and paste your system information.

@alikon Please tested exactly as described, with localized multilingual data, if it is really not an issue when using PostgreSQL (PDO).

avatar richard67 richard67 - change - 9 Apr 2020
Status New Confirmed
Build staging 4.0-dev
avatar richard67 richard67 - change - 9 Apr 2020
Category Updating
avatar richard67 richard67 - change - 9 Apr 2020
Labels Added: J4 Issue
avatar richard67 richard67 - labeled - 9 Apr 2020
avatar infograf768
infograf768 - comment - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
Priority Medium Urgent
avatar richard67
richard67 - comment - 9 Apr 2020

So MySQL 5.7, like in my case, too.

avatar richard67 richard67 - change - 9 Apr 2020
Title
[4.0] Updates from 3.10 broken for multilingual sites with MySQL (or MariaDB) databases
[4.0] Updates from 3.10 broken for multilingual sites with MySQL (and MariaDB?) databases
avatar richard67 richard67 - edited - 9 Apr 2020
avatar infograf768
infograf768 - comment - 9 Apr 2020

Also note these 2 errors when updating
[09-Apr-2020 08:08:34 UTC] PHP Notice: Undefined property: Joomla\CMS\Table\Extension::$locked in /Applications/MAMP/htdocs/joomla-cms-3.10-dev/libraries/src/Installer/InstallerAdapter.php on line 1155
[09-Apr-2020 08:08:35 UTC] PHP Warning: count(): Parameter must be an array or an object that implements Countable in /Applications/MAMP/htdocs/joomla-cms-3.10-dev/plugins/installer/override/override.php on line 134

avatar richard67
richard67 - comment - 9 Apr 2020

1st one comes from #28462 , 2nd one could be older.

avatar richard67
richard67 - comment - 9 Apr 2020

But these are separate issues.

avatar alikon
alikon - comment - 9 Apr 2020

confirm the warning about plugins/installer/override
@richard67 confirm tested as described

avatar richard67
richard67 - comment - 9 Apr 2020

@richard67 confirm tested as described

Do you mean you can confirm the issue with PostgreSQL, too? Or do you mean you can confirm what I wrote, that the issue is not there with PostgreSQL? It is not clear to me from your comment.

avatar alikon
alikon - comment - 9 Apr 2020

i confirm that the issue is not there with PostgreSQL (PDO)

avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
Title
[4.0] Updates from 3.10 broken for multilingual sites with MySQL (and MariaDB?) databases
[4.0] Updates from 3.10 broken for multilingual sites using the MySQLi database driver
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67 richard67 - change - 9 Apr 2020
The description was changed
avatar richard67 richard67 - edited - 9 Apr 2020
avatar richard67
richard67 - comment - 9 Apr 2020

With debug switched on in global config:

j4-update-error-web-assets

But if it's this, why doesn't it happen with any other db driver than MySQLi? Am maximum confused now.

avatar richard67
richard67 - comment - 9 Apr 2020

Ah there is another exception in the stack:

Joomla\Database\Exception\PrepareStatementFailureException: Unknown column 'm.publish_up' in 'where clause' at /home/richard/lamp/public_html/joomla-cms-3.10-dev/libraries/vendor/joomla/database/src/Mysqli/MysqliStatement.php:141 at Joomla\Database\Mysqli\MysqliStatement->__construct(object(mysqli), 'SELECT m.id,m.menutype,m.title,m.alias,m.note,m.link,m.type,m.level,m.language,m.browserNav,m.access,m.params,m.home,m.img,m.template_style_id,m.component_id,m.parent_id,m.path AS route,e.element AS componentFROM j3ux0_menu AS mLEFT JOIN j3ux0_extensions AS e ON m.component_id = e.extension_idWHERE ((m.published = 1 AND m.parent_id > 0 AND m.client_id = 0) AND (m.publish_up IS NULL OR m.publish_up <= ?)) AND (m.publish_down IS NULL OR m.publish_down >= ?)ORDER BY m.lft') (/home/richard/lamp/public_html/joomla-cms-3.10-dev/libraries/vendor/joomla/database/src/Mysqli/MysqliDriver.php:1048) at Joomla\Database\Mysqli\MysqliDriver->prepareStatement('SELECT m.id,m.menutype,m.title,m.alias,m.note,m.link,m.type,m.level,m.language,m.browserNav,m.access,m.params,m.home,m.img,m.template_style_id,m.component_id,m.parent_id,m.path AS route,e.element AS componentFROM j3ux0_menu AS mLEFT JOIN j3ux0_extensions AS e ON m.component_id = e.extension_idWHERE ((m.published = 1 AND m.parent_id > 0 AND m.client_id = 0) AND (m.publish_up IS NULL OR m.publish_up <= :currentDate1)) AND (m.publish_down IS NULL OR m.publish_down >= :currentDate2)ORDER BY m.lft') (/home/richard/lamp/public_html/joomla-cms-3.10-dev/libraries/vendor/joomla/database/src/DatabaseDriver.php:1891) at Joomla\Database\DatabaseDriver->setQuery(object(MysqliQuery)) (/home/richard/lamp/public_html/joomla-cms-3.10-dev/libraries/src/Menu/SiteMenu.php:155) at Joomla\CMS\Menu\SiteMenu->Joomla\CMS\Menu{closure}()
....

Prepared statement ... that fits better to db stuff.

avatar richard67
richard67 - comment - 10 Apr 2020

Meanwhile I think the above posts just show secondary errors but not the root cause. Of course prepared statements will fail in some PHP running after the database update when the database update hasn't run any SQL statement to add the publish up and down columns to the menu table.

avatar richard67
richard67 - comment - 10 Apr 2020

Can be also reproduced when using French only, without Persian.

avatar infograf768
infograf768 - comment - 10 Apr 2020

Let’s also test by not adding localised content, i.e with only the languagefilter plugin and with/without the langswitcher module. Can’t right now because I am on Ipad.

avatar richard67
richard67 - comment - 10 Apr 2020

@infograf768 sure, test it. I think I have tested that before, and the issue did not happen, but to be sure test again because I am not 100% sure now anymore.

At the end it will turn out that it's only happening when using such exotic languages like French, but not with normal languages like Chinese, Farsi, Arab, Klingon ... so maybe we just drop the support for French in J4 .. shall they learn English or German ;-) (teasing jokes)

avatar richard67
richard67 - comment - 10 Apr 2020

I did a database export before and after the update and there was no difference in structure or content coming form any of the update sql scripts ... only the update site info and so on, available updates, last login (because I had logged out in between), only such stuff has changes.

avatar richard67
richard67 - comment - 10 Apr 2020

@infograf768 I've just tested with installing French and enabling the multilanguage function and disabling localized content. Same result as with localized content. But I am sure from past tests when I did a monolanguage installation and after that logged in to the backend, installed languages and published content languages and enabled the language filter, then all was fine. But not 100% sure.

avatar richard67
richard67 - comment - 10 Apr 2020

@infograf768 Interesting: Now I have done everything to reproduce the issue, i.e. made multilingual installation with French and enabled multilanguage feature and enabled localized content. But before I did the Joomla Update, I deactivated the language filter plugin. And now it worked!!! Does that help you with having an idea what could go wrong where?

Or is maybe something with that saving or resetting the session? => @wilsonge ?

avatar wilsonge
wilsonge - comment - 10 Apr 2020

Figured out the issue. So when we have the language filter plugin enabled we always load the site menu to attach the build rules https://github.com/joomla/joomla-cms/blob/4.0-dev/plugins/system/languagefilter/languagefilter.php#L158

The problem is when we hit the login prompt we haven't run any of the SQL yet (this is normally the case if you check a Joomla update when you log in after performing the file update and is why you get the invalid template message).

The issue is that loading the site menu preloads all the items in the site menu. Normally this isn't a major issue. However we come to one of the new J4 features: Allowing publish up/down of menu items. https://github.com/joomla/joomla-cms/blob/4.0-dev/administrator/components/com_admin/sql/updates/mysql/4.0.0-2019-03-30.sql

The problem here is that we are now loading the site menu items (including the publish up/down columns before the column has been created!)

This leads to the exception that @richard67 mentioned here #28614 (comment)

The fact that postgres doesn't error when you try and filter by a non-existent column is interesting (and I'd claim a bug in postgres :) )

avatar richard67
richard67 - comment - 10 Apr 2020

@wilsonge Any idea yet how we shall solve it? And who shall do it? I am available and would work on it if nobody else does.

avatar richard67
richard67 - comment - 10 Apr 2020

@wilsonge With MySQL (PDO) it's as strange as with PostgreSQL (PDO), so it seems to be a PDO issue that there is no problem with missing columns.

avatar richard67
richard67 - comment - 10 Apr 2020

@wilsonge Would it help to add these columns to the menu table with an update sql in 3.10 so we can assume we have them when updating to 4, assuming that anyone who updates a 3.9 or earlier with Upload&Update is doing that on own risk?

Update: Alternatively we could add them in PHP in the update procedure before that login step which makes it fail then?

avatar wilsonge
wilsonge - comment - 11 Apr 2020

FWIW @infograf768 i don't think this is specific to j4 - it would happen in j3 if we did any sort of change to the menu table :(

avatar wilsonge
wilsonge - comment - 11 Apr 2020

Try this one #28635 - this also has the advantage that in the majority of the administrator client after patch in j4 you should see at least 1 less query because we no longer load all the site menu items unless we actually start to use the site router to build links.

avatar alikon alikon - change - 11 Apr 2020
Status Confirmed Closed
Closed_Date 0000-00-00 00:00:00 2020-04-11 06:41:11
Closed_By alikon
avatar alikon alikon - close - 11 Apr 2020
avatar alikon
alikon - comment - 11 Apr 2020

please test #28635

avatar wilsonge wilsonge - change - 11 Apr 2020
Labels Removed: ?
avatar wilsonge wilsonge - unlabeled - 11 Apr 2020

Add a Comment

Login with GitHub to post a comment