At a guess I'd say the tokeniser is running out of memory. This might fix the issue: #16621

@N8Solutions can you please test #16621?

@chrisdavenport & @franz-wohlkoenig

Thanks for the reply. That PR is for the Joomla v3.9 branch. I'm working with Joomla v3.8.12 and there is no /libraries/joomla/cache/ folder so the only file I can modify is administrator/components/com_finder/helpers/indexer/helper.php which I don't think by itself will help the issue.

Add ‘src’ After /libraries/ for the file locations

@tonypartridge With the help of a friend I modified the cache files in /libraries/src/Cache to be consistent with Joomla v3.8 and added /libraries/src/Cache/Storage/memory.php. The PR does not create any new problems but it also does not fix the original problem.

The original memory problem still persists.

I'm adding the changed files so you can see the changes I made. Just change the file extension to .php from .txt. The memory.php file was just a copy paste as it is written in the PR.
Cache.txt
CacheStorage.txt
helper.txt

I have added my tests and experience with this problem on issue #22220
The problem presents also in indexing both via the Smart Search component page and via the CLI, tried any possible setting for Memory Table Limit, Batch Size, and max heap table size on DB.

Today has been released 3.8.13 with no fix for this issue, and it seems there will be no more releases before the 3.9.

Since I necessarily had to find a way to re-enable search engine, I found that in the 3.8.12 all the indexer section has been rewritten, this is what has been done: #12253
So this work on the indexer code has introduced the issue we are experiencing in saving large articles.

As a temporary fix, I copied from the 3.8.11 full package all the files in administrator\components\com_finder\helpers\indexer\ to my current installation, overwriting the ones changed by the 3.8.12. Now everything works again as a charm: saving articles, indexing and CLI indexing as well.

I suggest to the people experiencing this issue to revert to the 3.8.11 indexer files until someone will fix any error introduced by the indexer rewriting released with the 3.8.12 .

Ill try and take a look at some point soon

Many thank
On 9 Oct 2018, 16:13 +0100, dagalumin notifications@github.com, wrote:

Today has been released 3.8.13 with no fix for this issue, and it seems there will be no more releases before the 3.9.
Since I necessarily had to find a way to re-enable search engine, I found that in the 3.8.12 all the indexer section has been rewritten, this is what has been done: #12253
So this work on the indexer code has introduced the issue we are experiencing in saving large articles.
As a temporary fix, I copied from the 3.8.11 full package all the files in administrator\components\com_finder\helpers\indexer\ to my current installation, overwriting the ones changed by the 3.8.12. Now everything works again as a charm: saving articles, indexing and CLI indexing as well.
I suggest to the people experiencing this issue to revert to the 3.8.11 indexer files until someone will fix any error introduced by the indexer rewriting released with the 3.8.12 .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I've taken a look @dagalumin and cannot see anything obvious to what you issue would be. The only possible additional overhead from my glance would be the else query in;

administrator/components/com_finder/helpers/indexer/indexer.php

can you try the removal of the query to see if that helps?

But I've tested on 3 Joomla! installs with different dB versions / types and we are getting a full index of that article with the full text provided and then doubled in amount again within a few seconds without issue and a memory limit of 128mb.

Thank you @tonypartridge

Removing the else query in the administrator/components/com_finder/helpers/indexer/indexer.php fixes the problem.

For clarification I removed the following from that file which spans lines 535 to 548.

		else
		{
			$query->values(
				$db->quote($tokens->term) . ', '
				. $db->quote($tokens->stem) . ', '
				. (int) $tokens->common . ', '
				. (int) $tokens->phrase . ', '
				. $db->escape((float) $tokens->weight) . ', '
				. (int) $context . ', '
				. $db->quote($tokens->language)
			);
			++$values;
		}
		$db->setQuery($query)->execute();

The reason I removed all of that is because I compared the old file and the new file and all of that was new for that section. If corrections should be made please advise.

I have successfully tested it on 2 sites now.

@dagalumin can you confirm?

kind regards

Well, if you really removed the $db->setQuery($query)->execute(); line then you in-effect no-opped that entire method which is writing data to the #__finder_tokens table because you removed the line that actually executes the database query.

@mbabker Thank you for pointing that out. I wasn't paying attention as I simply compared the old file with the new one and saw that everything on those lines was "new" from version 3.8.11.

If I add only that line back then it fails but this time with the following error message:

Save failed with the following error: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1

So removing the else statement there doesn't solve anything.

you need to move $db->setQuery($query)->execute(); before } it's executing even though it doesn't need to.

@tonypartridge Would you please do me the favor and post the code for the section like I did above showing exactly how it should look like without the else query and where the $db->setQuery($query)->execute(); should be placed?

change;

	// Iterate through the tokens to create SQL value sets.
	if (!is_a($tokens, 'FinderIndexerToken'))
	{
		foreach ($tokens as $token)
		{
			$query->values(
				$db->quote($token->term) . ', '
				. $db->quote($token->stem) . ', '
				. (int) $token->common . ', '
				. (int) $token->phrase . ', '
				. $db->escape((float) $token->weight) . ', '
				. (int) $context . ', '
				. $db->quote($token->language)
			);
			++$values;
		}
	}
	else
	{
		$query->values(
			$db->quote($tokens->term) . ', '
			. $db->quote($tokens->stem) . ', '
			. (int) $tokens->common . ', '
			. (int) $tokens->phrase . ', '
			. $db->escape((float) $tokens->weight) . ', '
			. (int) $context . ', '
			. $db->quote($tokens->language)
		);
		++$values;
	}
	$db->setQuery($query)->execute();

to

	// Iterate through the tokens to create SQL value sets.
	if (!is_a($tokens, 'FinderIndexerToken'))
	{
		foreach ($tokens as $token)
		{
			$query->values(
				$db->quote($token->term) . ', '
				. $db->quote($token->stem) . ', '
				. (int) $token->common . ', '
				. (int) $token->phrase . ', '
				. $db->escape((float) $token->weight) . ', '
				. (int) $context . ', '
				. $db->quote($token->language)
			);
			++$values;
		}
		$db->setQuery($query)->execute();
	}

@tonypartridge Thank you for clarifying and trying to help. I have implemented your suggestion however the problem remains.

I have just taken a quick look at it

i think that 2 bugs were introduced (please confirm as i am ill with fever lol)

• single FinderIndexerToken object passing to the method (please see methods signature / description) will no longer work
• chunking (to limit the size of the values clause) seems to be was removed

The removal of chunking will cause unlimited length query
thus depending on Mysql server configuration (like max_allowed_packet) you will have problems

Ok

only chunking was removed,
the single object check seems to be still there ? (the else case mentioned above)

895a148#diff-4960e3c19acee6699d798205cb91b590

@tonypartridge Thank you for clarifying and trying to help. I have implemented your suggestion however the problem remains.

Sorry for late answer, I tried the suggested fix and I confirm as @N8Solutions that also in my installation the problem remains.

I am happy to provide admin/ftp access, or the full package, if you want to investigate.
It's a very basic installation with just Akeeba backup and admintools and Showtime image gallery components installed. Just one article created with 12k words that causes the problem, both in local and on my dedicated Centos 6.9/Apache 2.2.15/PHP 7.0.32/MySql 5.5.61 server. Maybe it's not a high end machine, but doesn't explain why the problem is not happening with Joomla 3.8.11 version of the indexer.
Thanks all for the help.

That’s a very good point and explains the differences of the servers as to why it won’t causes an issue on my servers.
On 10 Oct 2018, 18:44 +0100, Georgios Papadakis notifications@github.com, wrote:

Ok
only chunking was removing, the single object check is still there
895a148#diff-4960e3c19acee6699d798205cb91b590
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Yes your servers can afford both version of the indexer, I just don't understand why the new version is so much more resource demanding... the rewriting has been done to make it lighter and faster, at least in the intentions.

It’s as Gerogios points out;

The removal of chunking will cause unlimited length query
thus depending on Mysql server configuration (like max_allowed_packet) you will have problems

It’s causing an unlimited length whereas on tighter constrained servers this wouldn’t work because of the size of the article. So really it needs splitting
On 10 Oct 2018, 19:46 +0100, dagalumin notifications@github.com, wrote:

Yes your servers can afford both version of the indexer, I just don't understand why the new version is so much more resource demanding... the rewriting has been done to make it lighter and faster, at least in the intentions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@N8Solutions
@dagalumin
Can you test if PR #22578 solves this issue ?

@ggppdk Σας ευχαριστώ πάρα πολύ!

Thank you very much for looking in to this! The Pull request works.

@N8Solutions
@dagalumin
Can you test if PR #22599 still solves this issue?
I took over the resolving of the issue from @ggppdk as I had some ideas to also optimize some things.
Turns out, I couldn't do that without breaking backwards compatibility.
However, I included some changes that @ggppdk proposed.
Thanks

Set to "closed" on behalf of @Quy by The JTracker Application at issues.joomla.org/joomla-cms/22098

Closing per PR #22599

Better late than never. I didn't see the final request to check if it was working or not but all is working as expected as you already know. Thank you again for those that worked on this issue.