[#17390] - Chunk the tokens array to avoid making extremely long queries.
- Fixed in Code Base
- 18 Dec 2017
- Medium
- Build: staging
- # 17390
- Diff
- okonomiyaki3000:chunk-tokens
User tests: Successful: Unsuccessful:
Pull Request for Issue # .
Summary of Changes
Chunk the tokens array (which can potentially be very long) before using it to generate a query. Run a query for each chunk instead of the whole array.
Testing Instructions
- Check that your php
max_execution_timeis something sensible like 30 seconds. - Turn on finder indexing for content.
- Create a super long article. Maybe like 10,000 words. Maybe this can help: http://slipsum.com/
- Save the article.
Expected result
Well, I expect the article to save properly and the page to reload without crashing.
Actual result
Most likely, the article will save but in the post-save indexing process there will be a timeout. This happens because we are looping over a huge array of tokens and adding each one to the values array of the query object. The query object handles this rather inefficiently (should be addressed in a different PR) but is mostly fine until the values array gets very large.
One solution is to avoid getting a huge values array by sending several smaller queries instead of one big one. This is that solution.
Documentation Changes Required
nope
joomla-cms-bot
- | Category | ⇒ | Administration com_finder |
| Status | New | ⇒ | Pending |
| Labels |
Added:
?
|
||
| Status | Pending | ⇒ | Needs Review |
Rebased with latest staging. I can't imagine why that drone test is failing. There's nothing in this code that would cause it to fail.
Hmm. So, is this getting in? Because this is a legit problem that people in the real world may actually face.
If it gets tested, sure. Looking at this again reminded me of something though.
There's already a similar (yet less elegant) implementation of this in the SQL Server subclass and I think it essentially is the same thing, so it looks like the overriding method can be removed.
Yeah, that's weird. Someone noticed this could happen with SQL Server but didn't think it would ever be a problem for other dbs? Anyway, I can remove that override and include it in this PR if you want.
That was me. SQL Server actually has (had? this was SQL Server 2008) a hard query limit of inserting 1000 values in a query which is why I had to do the override for that driver.
Rebased with latest staging and went ahead and removed that redundant override.
Just need one more approval...
I have tested this item
Tested and did code review too,
i am using similar code for updating DB in a faster way,
also this is not only about updating DB when having a long article,
there should be a considerable performance gain for the indexer in big sites
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/17390.
I have tested this item
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/17390.
| Status | Needs Review | ⇒ | Ready to Commit |
Ready to Commit after two successful tests.
| Status | Ready to Commit | ⇒ | Fixed in Code Base |
| Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2017-12-18 03:36:41 |
| Closed_By | ⇒ | mbabker | |
| Labels |
Added:
?
|
||
Status is set on "Needs Review".