[#36749] - [4.2] Smart Search: Prevent "#__finder_tokens full" error
- Fixed in Code Base
- 6 Jul 2022
- Medium
- Build: 4.0-dev
- # 36749
- Diff
- Hackwar:4.2-finder-memorytable
User tests: Successful: Unsuccessful:
Summary of Changes
Smart Search has a long standing issue that the #__finder_tokens table can run full and abort the indexing process. That table is of type "MEMORY", which can run full and there is an option to reduce the number of tokens to store in that table, which unfortunately doesn't work in all cases. The MEMORY table is used since it is a lot faster than a normal InnoDB table and since this is just used for temporary storage, writing speed is the main factor here.
This change pushes the code of addTokenToDb to the tokenizeToDbShort() method and better counts the number of tokens written to DB. It also removes the option from the Smart Search configuration and instead reads from the database what it is capable of. This most likely isn't a 100% solution, since there are edge cases where this could still fail, but it should improve the situation greatly.
This removes the addTokenToDb method, but since this is a protected method in a class of a component, this would still be in accordance to our backwards compatibility promise. Last but not least, this is remade from the PR #35995.
Testing Instructions
- Get a very long text and insert it into an article.
- When saving, see it fail with the error message from the title.
- Apply this PR.
- Save again and see it pass this time.
| Status | New | ⇒ | Pending |
joomla-cms-bot
- | Category | ⇒ | Administration com_finder |
| Title |
|
||||||
Damn, I was hoping nobody would notice. I accidentally based this PR on 4.0-dev instead of 4.1-dev and since the latest changes from 4.0-dev haven't been merged upwards, I tried to get around the bigger diff for now by selecting the 4.0-dev branch. I'll select the 4.1-dev branch.
"I was hoping nobody would notice" does not work with me
| Labels |
Added:
?
|
||
| Category | Administration com_finder | ⇒ | Administration com_categories com_contact com_content com_fields com_finder com_installer com_joomlaupdate com_templates Layout Libraries Front End Plugins |
| Labels |
Added:
?
Removed: ? |
||
| Category | Administration com_finder com_categories com_contact com_content com_fields com_installer com_joomlaupdate com_templates Layout Libraries Front End Plugins | ⇒ | Administration com_finder |
| Category | Administration com_finder | ⇒ | Administration com_finder Language & Strings |
| Labels |
Added:
Language Change
|
||
It partially works :)
It works for 100 paragraphs https://loremipsum.io/generator/?n=100&t=p
But fail for 500 paragraphs https://loremipsum.io/generator/?n=500&t=p
(Warning: do not try more, or your browser will explode
Not sure if it success ;)
Lets call it an improvement, which is why I would still consider this a success.
I have tested this item
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.
Shouldn't the parameter be removed from the params column in the extensions table in database for new installations here
https://github.com/joomla/joomla-cms/blob/4.1-dev/installation/sql/mysql/base.sql#L173
and here
https://github.com/joomla/joomla-cms/blob/4.1-dev/installation/sql/postgresql/base.sql#L179
?
@brianteeman What do you think? Should we do that here or leave it to a general cleanup for your issue #37386 ?
P.S. to my previous comment: @Hackwar If you want to fix the SQL with this PR, too, the best is that you just save the parameters after having made a new, clean installation with this PR applied, and then copy the value from database into the insert statement in base.sql.
I have tested this item
Saving an article with 23000 words gives me the message that the table is full. But after aplying the patch it still gives me the message that the table is full.
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.
@richard67 always do it when you see it. Another pr may not happen
This pull request has automatically rebased to 4.2-dev.
This pull requests has been automatically converted to the PSR-12 coding standard.
| Labels |
Added:
?
|
||
| Labels |
Added:
?
|
||
| Title |
|
||||||
| Category | Administration com_finder Language & Strings | ⇒ | Administration com_finder Language & Strings SQL Installation Postgresql |
Ok, I fixed a few more issues. @superknutsel , @Fedik would you be willing to test this again? I think I finally got this issue solved for good. Would be nice if we could get this into 4.2 still.
I have tested this item
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.
Still works for me, tried with ~33k words
I tried it with "War and Peace" from Tolstoi and while it failed to store the whole text, (because the input was too large) about a quarter of it was saved and then properly indexed.
I have tested this item
Hi, this time I tested it successfully.
Thanks for this.
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.
| Status | Pending | ⇒ | Ready to Commit |
RTC
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/36749.
| Labels |
Added:
?
Removed: ? |
||
Not sure what this PR fixes because for me "War and Peace" with 597.373 words worked before and afterwards but at least we have an option less YaY.
PS: tinymce is not optimized for writing books^^
| Status | Ready to Commit | ⇒ | Fixed in Code Base |
| Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2022-07-06 10:18:10 |
| Closed_By | ⇒ | HLeithner |
@HLeithner Can you please add the milestone Joomla! 4.2.0?
@Hackwar The title says "[4.1]" but the target branch is 4.0-dev. What's right now?