No Code Attached Yet bug
avatar hehemrin
hehemrin
13 Jul 2023

Steps to reproduce the issue

For example execute update of Joomla from release eg 4.3.2 to 4.3.3.

Expected result

  • Site updated.
  • Joomla states no new update is available.

Actual result

  • Site updated.
  • Joomla states error message:
    "1104 The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT is okay"

System information (as much as possible)

Joomla 4.3.3

Additional comments

Many configuration values is set by the owner of the eg LAMP-stack, which often is a service provider and not the web site responsible. The parameter "MAX_JOIN_SIZE" has default value 18446744073709551615 and the related parameter "SQL_BIG_SELECTS" has default value 1 (yes). If MAX_JOIN_SIZE is decreased, SQL_BIG_SELECTS is automatically changed to 0, but can manually be changed back to 1.

But the service provider may refuse or cannot do it. And before coming to the conclusion, this error requires investigation that can tentatively in my testing be avoided by update of the Joomla code. Instead of permanently change of the value(s) it is possible from the Joomla side to change SQL_BIG_SELECTS to 1 for each session. If SQL_BIG_SELECTS is set to 1, max_join_size will be ignored. A trouble is that when I edit the Joomla file myself as web site responsible the change is overwritten when Joomla is updated. I therefore suggest that Joomla add the code below or any better version, which seems to do the job.

Proposed addition to Joomla code base

MysqliDriver.php in folder
/libraries/vendor/joomla/database/src/Mysqli/
row 334 states (in the file revision valid for Joomla 4.3.3):
$this->options['sqlModes'] = explode(',', $this->setQuery('SELECT @@SESSION.sql_mode;')->loadResult());
Insert a new row after row 334:
mysqli_query($this->connection,"SET SQL_BIG_SELECTS = 1;" );

Note: more driver files may need to be updated in same manner, I have only focused on my setup.

References and further information

This issue has been discussed both in Joomla forum and Issue tracker. Here are some but not all related references:
Discussion incl above solution:
https://forum.joomla.org/viewtopic.php?f=811&t=995872
The trouble that new release change back the edited file:
https://forum.joomla.org/viewtopic.php?f=9&t=1002685
Similar Pending in issue tracker: https://issues.joomla.org/tracker/joomla-cms/17679
There are also a few closed related issues.
MariaDB knolwledge base:
https://mariadb.com/kb/en/server-system-variables/#max_join_size

Votes

# of Users Experiencing Issue
2/2
Average Importance Score
4.00

avatar hehemrin hehemrin - open - 13 Jul 2023
avatar hehemrin hehemrin - change - 13 Jul 2023
Labels Removed: ?
avatar joomla-cms-bot joomla-cms-bot - change - 13 Jul 2023
Labels Added: No Code Attached Yet
avatar joomla-cms-bot joomla-cms-bot - labeled - 13 Jul 2023
avatar richard67
richard67 - comment - 18 Jul 2023

@hehemrin Have you checked that with both the "MySQLi" and the "MySQL (PDO)" database types? If you have tested with MySQLi only, could you check in global configuration if the database type "MySQL (PDO)" is also available for you in the "server type" dropdown because the PDO extension of PHP is available, and if yes, change to that"MySQL (PDO)" driver and check again?

The right place to fix that is not here but in the database repository of the framework. There is already a pull request proposed for this issue: joomla-framework/database#266 .

However, it's not clear yet if it needs the same fix also for the "MySQL (PDO)" driver. If that is not the case and you can use that driver, it would be a usable workaround for you.

avatar richard67
richard67 - comment - 18 Jul 2023

See also #39479 .

avatar hehemrin
hehemrin - comment - 18 Jul 2023

@richard67 Thanks, yes it appears to be the same as joomla-framework/database#266. I did not find it in the Joomla Issue tracker when I searched. I am sorry that I classified it wrongly and maybe with wrong tool. I did what I thought was correct. Should I change this one in any way to correct classification?

MySQL (PDO) is available. I have not tested. But I am hesitant to test, because Joomla help text states (my translation): Do not change this to another data base technology, eg from MySQLi (MySQL-data base technology) to PostgreSQL (Postgres data base technology). That will damage your page." Now your proposal is within MySQL - can I simply flip between MySQLi and MySQL (PDO) back and forth without any trouble? If so, I can investigate it.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41156.

avatar richard67
richard67 - comment - 18 Jul 2023

Now your proposal is within MySQL - can I simply flip between MySQLi and MySQL (PDO) back and forth without any trouble?

Yes. You can do that in backend in global configuration. Before the changes are saved, the connection is tested with the new value, and if that fails you will see an error message, and the change is not saved and the old value is still used. And if that would fail for some reason you could still fall back to the previous value by editing file configuration.php.

avatar hehemrin
hehemrin - comment - 18 Jul 2023

Investigation:
I changed from MySQLi to MySQL (PDO). The change was accepted (saved). But when I go to the page "Check for Joomla update", I get a similar error message as for MySQLi: "42000, 1104, The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT is okay" (The difference is that I now also have the additional error code 42000).

So, this potential work around did not work, hence I changed back to MySQLi.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41156.

avatar richard67
richard67 - comment - 18 Jul 2023

@hehemrin Thanks for testing. To be honest, the result is what I expected. I only wasn't able to test that myself without much effort. So we know now that it needs the changes also for the MySQL (PDO) driver. I'll post in the PR for the framework.

avatar richard67
richard67 - comment - 22 Jul 2023

We are working on it, see also joomla-framework/database#284 and joomla-framework/database#285 .

avatar Hackwar Hackwar - change - 24 Aug 2023
Labels Added: bug
avatar Hackwar Hackwar - labeled - 24 Aug 2023
avatar jankorshavn63
jankorshavn63 - comment - 1 Sep 2023

Any updates to this issue yet? Just found my site in the same problem when trying to update to the latest Joomla version today.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41156.

avatar richard67
richard67 - comment - 1 Sep 2023

@jankorshavn63 No updates yet. For the mean time, as a workaround, use the code change proposed in the description of this issue. It will be lost after the update so you have to do it again before the next update, and the final fix might look different, but for now that's the only way out until we have a fix. For a good fix it is not sufficient to just hard-code it as proposed here. Currently I'm working on a fix but it will take a bit time.

avatar jankorshavn63
jankorshavn63 - comment - 22 Nov 2023

As this seem to be a problem on several of my sites, I hope there will be a solution soon. What's the latest develoment on this issue?

avatar norwegian69
norwegian69 - comment - 1 Dec 2023

Also having this problem after my server provider replaced Cpanel with a new system.
Changing the code above works fine, but have many web pages so it becomes time consuming for each update of Joomla.

avatar richard67
richard67 - comment - 22 Dec 2023

@hehemrin @jankorshavn63 @norwegian69 I have prepared some pull requests which would fix the issue, but some maintainers are reluctant and say that the better fix would be to find and fix the critical SQL query or queries which exceed the max_join_size, and they might be right.

Unfortunately I could not reproduce the issue on a clean Joomla core using the max_join_size reported in the support forum, so it seems to need certain starting conditions, maybe a large number of 3rd party extensions.

So it would help if you could report back the actual value of max_join_size on the environment where you have the issue (can be found out e.g. in phpMyAdmin with SELECT @@SESSION.MAX_JOIN_SIZE;), and does it happen in your environment also with a clean Joomla core without any 3rd party extensions?

avatar norwegian69
norwegian69 - comment - 22 Dec 2023

The response from the server supplier was:
This is unfortunately due to a restriction on our server, but you can solve it by adding the following to Joomla's code:
MysqliDriver.php in folder
/libraries/vendor/joomla/database/src/Mysqli/
row 334 states (in the file revision valid for Joomla 4.3.3):
$this->options['sqlModes'] = explode(',', $this->setQuery('SELECT @@SESSION.sql_mode;')->loadResult());
Insert a new row after row 334:
mysqli_query($this->connection,"SET SQL_BIG_SELECTS = 1;" );
And he put a link to this discussion. :)

Not sure if I did this correctly but got this value: 67108864
see screenshot!
Screenshot 2023-12-22 at 13-39-16 proisp dbadmin one com _ mysql cu8eqrspl service one phpMyAdmin 5 2 1

avatar richard67
richard67 - comment - 22 Dec 2023

The response from the server supplier was:
...

@norwegian69 We know that already.

Not sure if I did this correctly but got this value: 67108864

That's the same value as it had been reported in the support forum. I have tried here to reproduce the issue with that value and sql_big_selects switched off, but I did not get the error. So it seems to need certain starting conditions like e.g. a certain number of 3rd party extensions.

Maybe it would help if I could download a backup (files zipped and database SQL export, or an Akeeba backup) of an affected site and test that here with the mentioned value. @hehemrin @jankorshavn63 @norwegian69 Could anyone of you provide a download of such a backup?

I you don't want to provide a public link you could send me the link with access information to my Joomla community email address richard<dot>fath<at>community<dot>joomla<dot>org (replace the <dot> and <at> by the corresponding characters). Or if you are on Mattemost in the Joomla Community, direct message me there.

avatar hehemrin
hehemrin - comment - 22 Dec 2023

Hi,
I @.***) use the following 3rd party extensions:

  • Akeeba backup
  • JCE Editor
  • CComment
  • OSMap incl Joomlashack Framework

Best regards
Henrik Hemrin

Den 2023-12-22 kl. 14:24, skrev Richard Fath:

The response from the server supplier was:
...

@norwegian69 https://github.com/norwegian69 We know that already.

Not sure if I did this correctly but got this value: 67108864

That's the same value as it had been reported in the support forum. I
have tried here to reproduce the issue with that value and
sql_big_selects switched off, but I did not get the error. So it seems
to need certain starting conditions like e.g. a certain number of 3rd
party extensions.

Maybe it would help if I could download a backup (files zipped and
database SQL export, or an Akeeba backup) of an affected site and test
that here with the mentioned value. @hehemrin
https://github.com/hehemrin @jankorshavn63
https://github.com/jankorshavn63 @norwegian69
https://github.com/norwegian69 Could anyone of you provide a download
of such a backup?

I you don't want to provide a public link you could send me the link
with access information to my Joomla community email address
|richardfathcommunityjoomlaorg| (replace the ||
and || by the corresponding characters). Or if you are on Mattemost
in the Joomla Community, direct message me there.


Reply to this email directly, view it on GitHub
#41156 (comment), or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5V4KLEUS7SOHZ6WBHZR4TYKWCXFAVCNFSM6AAAAAA2JPVF76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGY4DKMJUGI.
You are receiving this because you were mentioned.Message ID:
@.***>

avatar richard67
richard67 - comment - 23 Dec 2023

@hehemrin How many articles, categories and menu items has the site approximately?

avatar hehemrin
hehemrin - comment - 23 Dec 2023

Site data:

381 articles
12 categories
21 menu items

Sitemap:
https://www.hemrin.com/about-and-contact/sitemap-webbplatskarta?view=html&id=1

Site: https://www.hemrin.com/

I can add that I admin also another site for an org with slightly
different setup of extensions. At same hotel, same SQL_BIG_SELECTS
issue. The issue started for both when migrated from Jommla 3 to 4 this
2023 summer/autumn.

Prior migration, I explored J4 on a copy of my site as a local host on a
Linux laptop. Not exactly same versions of LAMP-softwares nor settings,
of course. I did not have the issue with SQL_BIG_SELECTS there, as I did
not have a limit on MAX_JOIN_SIZE.

Den 2023-12-23 kl. 14:01, skrev Richard Fath:

@hehemrin https://github.com/hehemrin How many articles, categories
and menu items has the site approximately?


Reply to this email directly, view it on GitHub
#41156 (comment), or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5V4KMVED5D3HH3LA2QEMLYK3I4HAVCNFSM6AAAAAA2JPVF76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGI4DSOJYHA.
You are receiving this because you were mentioned.Message ID:
@.***>

avatar richard67
richard67 - comment - 25 Dec 2023

I still can't get the error on a Joomla 4.4.2-dev with Akeeba Backup Core and OSMap Free installed when having sql_big_selects switched off and a small max_join_size of 300000 on my local mysql server.

I haven't tried yet with one of the other extensions which @hehemrin has mentioned (JCE Editor and CComment).

In a closed other issue about that problem, someone has posted a comment at the end that he has the problem, too, and has provided a stack trace. See this comment: #17580 (comment)

The stack trace shows that the query came from "administrator/components/com_virtuemart/helpers/vmmodel.php", which clearly is a 3rd party extension.

@hehemrin @norwegian69 @jankorshavn63 Could you provoke the error by reverting your workaround, switching the update source on the Joomla Update Component to "Custom URL" and use the URL https://update.joomla.org/core/nightlies/next_major_list.xml of the 5.0.2-dev nightly builds? If you can, then switch on "Debug System" in Global Configuration, and you should be shown a stack trace when that error happens. If so, then please post the stack trace here, similar to as it was done in the mentioned comment in the closed other issue.

I have prepared 2 pull requests for the CMS which would solve this issue, see #42557 or alternatively #42558 . But I will leave this issue open because 1. we still need to find out if there is a query from the CMS core which causes that issue, and 2. my proposed pull requests might be not accepted. But you could help by testing them.

avatar hehemrin
hehemrin - comment - 27 Dec 2023

Hi, in response to question by Richard Fath 25 Dec, I report as follows:

I copied my live site with Akeeba backup.
I installed this copy on a Linux Mint laptop.

Joomla version: 4.4.1

Localhost server settings:
PHP 8.1.2-1ubuntu2.14
MySQLi 10.6.12-MariaDB-0ubuntu0.22.04.1

I take away my live site work around:
MysqliDriver.php in folder
/libraries/vendor/joomla/database/src/Mysqli/
I delete my addition after row 334:
mysqli_query($this->connection,"SET SQL_BIG_SELECTS = 1;" );
This means the file is as default in Joomla.

I change values for the localhost server:
Max_join_size to 67 108 864
Sql_big_selects to OFF
These values corresponds to the settings on the live site server.

I now go to the localhost Joomla backend, and to the page for Joomla
updates. Instead of seeing update status, channel etc, I get this
familiar error message:
An error has occurred:
"1104 The SELECT would examine more than MAX_JOIN_SIZE rows; check your
WHERE and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT
is okay "

So, this means that I can replicate my live site issue on the localhost.

Next step is the test proposed by Richard Fath:

  • Switching the update source on the Joomla Update
    Component to "Custom URL" and use the URL
    https://update.joomla.org/core/nightlies/next_major_list.xml of the
    5.0.2-dev nightly builds
  • Switch on "Debug System" in Global Configuration, and you should be
    shown a stack
    trace when that error happens.
  • Post the stack trace here, similar to as it was done in the mentioned
    comment in the closed other issue.

I change update channel and also set lowest stability to Development
from Stable.

Now I go to the page for Joomla update. It looks as follows (Swedish
language settings):

========================================

Ett fel har inträffat

 1104 The SELECT would examine more than MAX_JOIN_SIZE rows; check 

your WHERE and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the
SELECT is okay

Call stack

Function Location

1 ()
JROOT/libraries/vendor/joomla/database/src/Mysqli/MysqliStatement.php:442
2 Joomla\Database\Mysqli\MysqliStatement->execute()
JROOT/libraries/vendor/joomla/database/src/DatabaseDriver.php:675
3 Joomla\Database\DatabaseDriver->execute()
JROOT/libraries/vendor/joomla/database/src/DatabaseDriver.php:1277
4 Joomla\Database\DatabaseDriver->loadColumn()
JROOT/libraries/src/Extension/ExtensionHelper.php:406
5 Joomla\CMS\Extension\ExtensionHelper::getCoreExtensionIds()
JROOT/administrator/components/com_joomlaupdate/src/Model/UpdateModel.php:1414
6
Joomla\Component\Joomlaupdate\Administrator\Model\UpdateModel->getNonCoreExtensions()
JROOT/libraries/src/MVC/View/AbstractView.php:152
7 Joomla\CMS\MVC\View\AbstractView->get()
JROOT/administrator/components/com_joomlaupdate/src/View/Joomlaupdate/HtmlView.php:164
8
Joomla\Component\Joomlaupdate\Administrator\View\Joomlaupdate\HtmlView->display()
JROOT/administrator/components/com_joomlaupdate/src/Controller/DisplayController.php:87
9
Joomla\Component\Joomlaupdate\Administrator\Controller\DisplayController->display()
JROOT/libraries/src/MVC/Controller/BaseController.php:693
10 Joomla\CMS\MVC\Controller\BaseController->execute()
JROOT/libraries/src/Dispatcher/ComponentDispatcher.php:143
11 Joomla\CMS\Dispatcher\ComponentDispatcher->dispatch()
JROOT/libraries/src/Component/ComponentHelper.php:361
12 Joomla\CMS\Component\ComponentHelper::renderComponent()
JROOT/libraries/src/Application/AdministratorApplication.php:143
13 Joomla\CMS\Application\AdministratorApplication->dispatch()
JROOT/libraries/src/Application/AdministratorApplication.php:186
14 Joomla\CMS\Application\AdministratorApplication->doExecute()
JROOT/libraries/src/Application/CMSApplication.php:293
15 Joomla\CMS\Application\CMSApplication->execute()
JROOT/administrator/includes/app.php:61
16 require_once() JROOT/administrator/index.php:32

Tidigare fel

The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE
and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT is okay
Call stack

Function Location

1 ()
JROOT/libraries/vendor/joomla/database/src/Mysqli/MysqliStatement.php:435
2 mysqli_stmt->execute()
JROOT/libraries/vendor/joomla/database/src/Mysqli/MysqliStatement.php:435
3 Joomla\Database\Mysqli\MysqliStatement->execute()
JROOT/libraries/vendor/joomla/database/src/DatabaseDriver.php:675
4 Joomla\Database\DatabaseDriver->execute()
JROOT/libraries/vendor/joomla/database/src/DatabaseDriver.php:1277
5 Joomla\Database\DatabaseDriver->loadColumn()
JROOT/libraries/src/Extension/ExtensionHelper.php:406
6 Joomla\CMS\Extension\ExtensionHelper::getCoreExtensionIds()
JROOT/administrator/components/com_joomlaupdate/src/Model/UpdateModel.php:1414
7
Joomla\Component\Joomlaupdate\Administrator\Model\UpdateModel->getNonCoreExtensions()
JROOT/libraries/src/MVC/View/AbstractView.php:152
8 Joomla\CMS\MVC\View\AbstractView->get()
JROOT/administrator/components/com_joomlaupdate/src/View/Joomlaupdate/HtmlView.php:164
9
Joomla\Component\Joomlaupdate\Administrator\View\Joomlaupdate\HtmlView->display()
JROOT/administrator/components/com_joomlaupdate/src/Controller/DisplayController.php:87
10
Joomla\Component\Joomlaupdate\Administrator\Controller\DisplayController->display()
JROOT/libraries/src/MVC/Controller/BaseController.php:693
11 Joomla\CMS\MVC\Controller\BaseController->execute()
JROOT/libraries/src/Dispatcher/ComponentDispatcher.php:143
12 Joomla\CMS\Dispatcher\ComponentDispatcher->dispatch()
JROOT/libraries/src/Component/ComponentHelper.php:361
13 Joomla\CMS\Component\ComponentHelper::renderComponent()
JROOT/libraries/src/Application/AdministratorApplication.php:143
14 Joomla\CMS\Application\AdministratorApplication->dispatch()
JROOT/libraries/src/Application/AdministratorApplication.php:186
15 Joomla\CMS\Application\AdministratorApplication->doExecute()
JROOT/libraries/src/Application/CMSApplication.php:293
16 Joomla\CMS\Application\CMSApplication->execute()
JROOT/administrator/includes/app.php:61
17 require_once() JROOT/administrator/index.php:32

=========================================

Best regards
Henrik Hemrin

Den 2023-12-25 kl. 19:20, skrev Richard Fath:

I still can't get the error on a Joomla 4.4.2-dev with Akeeba Backup
Core and OSMap Free installed when having sql_big_selects switched off
and a small max_join_size of 300000 on my local mysql server.

I haven't tried yet with one of the other extensions which @hehemrin
https://github.com/hehemrin has mentioned (JCE Editor and CComment).

In a closed other issue about that problem, someone has posted a comment
at the end that he has the problem, too, and has provided a stack trace.
See this comment: #17580 (comment)
#17580 (comment)

The stack trace shows that the query came from
"administrator/components/com_virtuemart/helpers/vmmodel.php", which
clearly is a 3rd party extension.

@hehemrin https://github.com/hehemrin @norwegian69
https://github.com/norwegian69 @jankorshavn63
https://github.com/jankorshavn63 Could you provoke the error by
reverting your workaround, switching the update source on the Joomla
Update Component to "Custom URL" and use the URL
https://update.joomla.org/core/nightlies/next_major_list.xml
https://update.joomla.org/core/nightlies/next_major_list.xml of the
5.0.2-dev nightly builds? If you can, then switch on "Debug System" in
Global Configuration, and you should be shown a stack trace when that
error happens. If so, then please post the stack trace here, similar to
as it was done in the mentioned comment in the closed other issue.

I have prepared 2 pull requests for the CMS which would solve this
issue, see #42557 #42557 or
alternatively #42558 #42558 .
But I will leave this issue open because 1. we still need to find out if
there is a query from the CMS core which causes that issue, and 2. my
proposed pull requests might be not accepted.


Reply to this email directly, view it on GitHub
#41156 (comment), or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5V4KN56NX4NDIJ4J6ASOLYLG7WBAVCNFSM6AAAAAA2JPVF76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRZGA3TMMJXGE.
You are receiving this because you were mentioned.Message ID:
@.***>

avatar richard67
richard67 - comment - 28 Dec 2023

@hehemrin Well, that stack traces help. I see now which query causes the issue. I just don't know yet why it doesn't do that on my MySQL 8 database with sql_big_selects = OFF and a small max_join_size. Maybe MariaDB 10.6 and MySQL behave differently, or fore some reason an index is missing in your database, maybe due to a long update history of that site. I don't really believe that the latter is the case, but just to be sure, could you execute following SQL statement in a tool like e.g. phpMyAdmin and report back the result? Replace the #__ by your actual table prefix before executing.

SHOW INDEXES IN `#__extensions` WHERE Key_name = 'extension';

The result should be 4 rows with the first 5 columns like this (the other columns are not of interest):

Table             Non_unique  Key_name   Seq_in_index  Column_name
j3ux0_extensions  1           extension  1             type
j3ux0_extensions  1           extension  2             element
j3ux0_extensions  1           extension  3             folder
j3ux0_extensions  1           extension  4             client_id

Do you get the same? Or is that index "extension" missing for some reason?

avatar richard67
richard67 - comment - 28 Dec 2023

Meanwhile I've set up a MariaDB 10.4 and there I can reproduce the issue. So it seems indeed that MySQL and MariaDB behave differently.

I will see if I can fix that query.

avatar alikon
alikon - comment - 28 Dec 2023

can you post the query ? very curious

avatar richard67
richard67 - comment - 28 Dec 2023

can you post the query ? very curious

@alikon Is is a bit too long. It's the one created here: https://github.com/joomla/joomla-cms/blob/4.4-dev/libraries/src/Extension/ExtensionHelper.php#L385-L409

A possible fix could be not to use prepared statements (bindArray) because the values are from a hard-coded array in that PHP file and not from any user input.

Another fix could be to use CONCAT to combine the values of the 4 columns to 1 for each row and then use that in the where clause in an IN condition.

As we already have the "extensions" index on the combination of these 4 columns, the issue is not caused by a missing index.

avatar richard67
richard67 - comment - 28 Dec 2023

@alikon The query looks like this (shortened):

SELECT `extension_id`
FROM `#__extensions`
WHERE (`type` = :preparedArray1 AND `element` = :preparedArray2 AND `folder` = :preparedArray3 AND `client_id` = :preparedArray4)
   OR (`type` = :preparedArray5 AND `element` = :preparedArray6 AND `folder` = :preparedArray7 AND `client_id` = :preparedArray8)
   OR (`type` = :preparedArray9 AND `element` = :preparedArray10 AND `folder` = :preparedArray11 AND `client_id` = :preparedArray12)
...
   OR (`type` = :preparedArray929 AND `element` = :preparedArray930 AND `folder` = :preparedArray931 AND `client_id` = :preparedArray932)

Changing from prepared statement to static SQL does not help:

SELECT `extension_id`
FROM `#__extensions`
WHERE (`type` = 'component' AND `element` = 'com_actionlogs' AND `folder` = '' AND `client_id` = 1)
   OR (`type` = 'component' AND `element` = 'com_admin' AND `folder` = '' AND `client_id` = 1)
   OR (`type` = 'component' AND `element` = 'com_ajax' AND `folder` = '' AND `client_id` = 1)
...
   OR (`type` = 'template' AND `element` = 'cassiopeia' AND `folder` = '' AND `client_id` = 0)
avatar richard67
richard67 - comment - 28 Dec 2023

@alikon Changing the query so that it uses the CONCAT function (which should work on PostgreSQL, too) and one single where condition IN would solve the issue here:

SELECT `extension_id`
FROM `#__extensions`
WHERE CONCAT(`type`,'|',`element`,'|',`folder`,'|',`client_id`) IN (
'component|com_actionlogs||1',
'component|com_admin||1',
'component|com_ajax||1',
...
'template|cassiopeia||0');
avatar alikon
alikon - comment - 28 Dec 2023

or if you add an index like (type , element, folder, client_id), i don't have a MariaDb istance right now to test

avatar richard67
richard67 - comment - 28 Dec 2023

or if you add an index like (type , element, folder, client_id), i don't have a MariaDb istance right now to test

@alikon I have written in my previous comment that we have that index already!

avatar richard67
richard67 - comment - 28 Dec 2023

I will prepare a PR with a fix sooner or later today.

avatar richard67
richard67 - comment - 28 Dec 2023

What can be is that for some reason MariaDB doesn't use the index in the original statement but uses it in my proposed fix, or the statement itself or its where clause are too long and will be much sorter with my proposal.

avatar richard67
richard67 - comment - 28 Dec 2023

My fix seems to work, also when using prepared statements (whereIn). Stay tuned for a PR.

avatar richard67
richard67 - comment - 28 Dec 2023

PR is #42576 . It is draft because I need to complete the testing instructions. As soon as ready for tests I will post here and close this issue. But interested readers may already have a look on it.

avatar alikon
alikon - comment - 28 Dec 2023

i've made a quick test with #42576 on Postgres, Mysql
using System -> Extensions filtering core and non core extensions same results as before the pr

avatar richard67
richard67 - comment - 28 Dec 2023

i've made a quick test with #42576 on Postgres, Mysql using System -> Extensions filtering core and non core extensions same results as before the pr

That filtering is something else. It doesn't seem to use the critical query. Please follow the testing instructions in my PR.

avatar richard67 richard67 - change - 28 Dec 2023
Status New Closed
Closed_Date 0000-00-00 00:00:00 2023-12-28 15:49:33
Closed_By richard67
avatar richard67 richard67 - close - 28 Dec 2023
avatar richard67
richard67 - comment - 28 Dec 2023

Closing as having a pull request, Please test #42576 . Thanks in advance.

avatar alikon
alikon - comment - 28 Dec 2023

sorry to contradict you but they use the same query

image

avatar richard67
richard67 - comment - 28 Dec 2023

@alikon Well that can be used as additional test. Thanks for the info.

avatar richard67
richard67 - comment - 29 Dec 2023

@alikon O've updated the testing instructions in my PR by the case you have mentioned. I'd be happy if you could find the time and test it. You can get a suitable MariaDB version e.g. with the latest XAMPP version.

avatar alikon
alikon - comment - 29 Dec 2023

test done with docker and Maria 10.6

avatar jankorshavn63
jankorshavn63 - comment - 12 Feb 2024

@jankorshavn63 No updates yet. For the mean time, as a workaround, use the code change proposed in the description of this issue. It will be lost after the update so you have to do it again before the next update, and the final fix might look different, but for now that's the only way out until we have a fix. For a good fix it is not sufficient to just hard-code it as proposed here. Currently I'm working on a fix but it will take a bit time.

I see that much have happened with this issue, but what's the current status now? I'm not that experienced and would appreciate any advice on how to get past this error when trying to upgrade multiple Joomle sites..

avatar richard67
richard67 - comment - 12 Feb 2024

@jankorshavn63 The query which has caused that issue in the CMS core has been identified and fixed in the last releases 4.4.2 and 5.0.3, see #42576 . However there was a mistake in it which will be fixed with the upcoming versions 4.4.3 and 5.0.3, see #42638 . You can apply the changes of these 2 PRs manually by editing the file "libraries/src/Extension/ExtensionHelper.php" and then the issue should be fixed.

avatar jankorshavn63
jankorshavn63 - comment - 26 Feb 2024

I updated one site from 4.4.2 to 4.4.3 today, but Joomla states that the upgrade went well BUT with error:

The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT is okay

The same error message appears if I reinstall Joomla core files..

Any advice?

avatar richard67
richard67 - comment - 26 Feb 2024

I updated one site from 4.4.2 to 4.4.3 today, but Joomla states that the upgrade went well BUT with error:

The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT is okay

The same error message appears if I reinstall Joomla core files..

Any advice?

@jankorshavn63 It can be that it happens one time after the update, but then it should not happen again with that critical query from the CMS core. Maybe you have some 3rd party extension which causes the same issue?

avatar jankorshavn63
jankorshavn63 - comment - 27 Feb 2024

Upgraded another site today, same error. And if I activate error reporting, I get this:

Warning: set_time_limit(): Cannot set max execution time limit due to system policy in /customers/3/0/9/jkweb.no/httpd.www/libraries/src/Filesystem/Folder.php on line 574

This site has almost no extensions yet, just starting to work on it...

avatar richard67
richard67 - comment - 27 Feb 2024

@jankorshavn63 Are you sure you‘ve updated from 4.4.3 to 5.0.3? I‘m asking because the error with the set_time_limit should also be fixed in both 4.4.3 and 5.5.3, see #41523 .

avatar richard67
richard67 - comment - 27 Feb 2024

Or from which to which version have you updated?

avatar jankorshavn63
jankorshavn63 - comment - 27 Feb 2024

Only using J4, so I updated today from 4.4.1 to 4.4.3.Vennlig hilsenJan KorshavnWeb: www.jkweb.no Tlf: 900 51 880Følg JKWEB på Facebook for praktiske tips og oppdateringer på Joomlas utvikling.27. feb. 2024 kl. 18:08 skrev Richard Fath @.***>:
@jankorshavn63 Are you sure you‘ve updated from 4.4.3 to 5.0.3? I‘m asking because the error with the set_time_limit should also be fixed in both 4.4.3 and 5.5.3, see #41523 .

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

avatar richard67
richard67 - comment - 27 Feb 2024

@jankorshavn63 As I meanwhile found out, the set_time_limit error has even been fixed with 4.4.0. So I‘m confused now.

The SQL error, do you get that only one time at the end of the update? Or do you get that always when visiting the update component in backend? Or if you get it in other situations, in which?

avatar jankorshavn63
jankorshavn63 - comment - 27 Feb 2024

In this last case I got it first when trying to update, then I added that line manually, ran the update perfectly but it ended with the error message I wrote earlier..Vennlig hilsenJan KorshavnWeb: www.jkweb.no Tlf: 900 51 880Følg JKWEB på Facebook for praktiske tips og oppdateringer på Joomlas utvikling.27. feb. 2024 kl. 18:55 skrev Richard Fath @.***>:
@jankorshavn63 As I meanwhile found out, the set_time_limit error has even been fixed with 4.4.0. So I‘m confused now.
The SQL error, do you get that only one time at the end of the update? Or do you get that always when visiting the update component in backend? Or if you get it in other situations, in which?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

avatar richard67
richard67 - comment - 27 Feb 2024

In this last case I got it first when trying to update

@jankorshavn63 That's expected because before the update the fix was not present yet.

then I added that line manually, ran the update perfectly but it ended with the error message I wrote earlier

One time at the end of the update that might also be expected.

Could you try something?

Go to System -> Extensions -> Manage to get the list of all extensions. Then use the filter for core extensions. Does the SQL error happen?

avatar jankorshavn63
jankorshavn63 - comment - 27 Feb 2024

I just tried that on 3 sites already updated to 4.4.3 with the same error after upgrading, nothing special happens. Joomla just lists all core extensions..

avatar richard67
richard67 - comment - 27 Feb 2024

I just tried that on 3 sites already updated to 4.4.3 with the same error after upgrading, nothing special happens. Joomla just lists all core extensions..

@jankorshavn63 So I think that SQL error is fixed and happens only once at the end of an update from a version where it was not fixed yet, and when the next update is available and applied it will not happen again.

avatar jankorshavn63
jankorshavn63 - comment - 27 Feb 2024

Tried another site, updated from 4.4.0 to 4.4.3, same error after upgrade (had to do the manual update of that file first). Screnshot added.
Screenshot 2024-02-27 at 20-23-56 Last opp og oppdater - Nettportalen for basketballdommere - Administrasjon

avatar richard67
richard67 - comment - 27 Feb 2024

@jankorshavn63 Did you read my previous comment?

avatar jankorshavn63
jankorshavn63 - comment - 27 Feb 2024

So I will have to wait until the next update to verify if the error is gone? For your info, here's the info from that log:

2024-02-27T19:22:57+00:00 ERROR 46.9.25.85 update An error has occurred while running "JoomlaInstallerScript::updateManifestCaches". Code: 1104. Message: The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET MAX_JOIN_SIZE=# if the SELECT is okay.
2024-02-27T19:23:15+00:00 WARNING 46.9.25.85 jerror Could not delete folder. Path: [ROOT]/administrator/cache/page

avatar richard67
richard67 - comment - 27 Feb 2024

So I will have to wait until the next update to verify if the error is gone?

Yes.

If the error happens again with the next update then please open a new issue. Thanks in advance.

For your info, here's the info from that log:

Thanks. It shows that it’s that error which has been fixed.

avatar jankorshavn63
jankorshavn63 - comment - 27 Feb 2024

Thx for your help (and patience)!

Add a Comment

Login with GitHub to post a comment