? ? Pending

User tests: Successful: Unsuccessful:

avatar laoneo
laoneo
13 Mar 2017

Pull Request for Issue #14473.

Summary of Changes

Right now when the values of the fields are fetched, for every field a query is done. This pr changes the behavior to one query for all fields.

Integer indexes are faster to lookup than string indexes, the context column got removed from the field_values table, we gain a bit of performance here as well.

Ping @Bakual and @mbabker for review.
Ping @alikon and @csthomas did I change the installer and upgrade scripts correctly?

Testing Instructions

  • Create some fields.
  • Edit an article.
  • Set some values for the fields.
  • Save the article.
  • View the article on the front.

Expected result

All of these should work as before, but with less queries. When the debugging is on, then you should see less queries logged.

Actual result

It works, but with more queries.

Documentation Changes Required

None.

avatar laoneo laoneo - open - 13 Mar 2017
avatar laoneo laoneo - change - 13 Mar 2017
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 13 Mar 2017
Category SQL Administration com_admin Postgresql MS SQL com_fields Installation Front End Plugins
avatar laoneo laoneo - change - 13 Mar 2017
Labels Added: ?
avatar laoneo
laoneo - comment - 13 Mar 2017

Thanks @csthomas removed the double null check.

avatar Bakual
Bakual - comment - 14 Mar 2017

After thinking about this a bit, the only question I have is if we really need both the "getFieldValue" and "getFieldValues" methods. They do quite the same just instead of using a "->where field_id IN (...)" instead of "->where field_id = ...".
That could also be done using the same method with a conditional based on the field_id parameter (multiple ids vs single integer).

avatar mbabker
mbabker - comment - 14 Mar 2017

Please don't combine the two. That violates SRP.

avatar Bakual
Bakual - comment - 14 Mar 2017

Stupid question, what is SRP?

avatar rdeutz
rdeutz - comment - 14 Mar 2017

single responsibility prinzip

avatar Bakual
Bakual - comment - 14 Mar 2017

Makes sense ?

So better to have a method to fetch the value(s) for a single field and one for fetching values for multiple fields allthough they do basically the same thing?
If that's preferred, then it is fine as it is. It was just something to consider ?

avatar mbabker
mbabker - comment - 14 Mar 2017

Right. Basically you want your methods to be responsible for one task and one task only. Fetching the data for a single field and fetching the data for multiple fields are two different tasks, will presumably have two different return types (one might be a value object and the other may be a collection of value objects), and may be post-processed by two different code paths (even if you're doing a foreach loop and it just calls the path for the single item, it still has to have that extra logic). So it's better from a structural standpoint to separate those out.

avatar laoneo
laoneo - comment - 15 Mar 2017

Changed the getValue function to use getValues and removed the not needed code as pointed out by @joomdonation. Additionally added some inline comments to explain what the function is doing.

avatar rdeutz
rdeutz - comment - 15 Mar 2017

@laoneo when you on it could you fix the code style issues

avatar laoneo
laoneo - comment - 15 Mar 2017

@rdeutz which ones? Travis is running trough.

avatar laoneo
laoneo - comment - 15 Mar 2017

Ah sorry, didn't realize that drone is now doing the CS stuff. Give me a minute.

c2af37b 15 Mar 2017 avatar laoneo CS
avatar laoneo
laoneo - comment - 15 Mar 2017

Can drone be restarted somehow?

avatar zero-24
zero-24 - comment - 15 Mar 2017

done @laoneo

avatar laoneo
laoneo - comment - 15 Mar 2017

@rdeutz all good now.

avatar laoneo
laoneo - comment - 20 Mar 2017

Should we also add here a release blocker label as it reduces the amount of SQL queries?

avatar mbabker
mbabker - comment - 20 Mar 2017

Added due to the major performance issues this PR aims to address.

avatar joomdonation joomdonation - test_item - 24 Mar 2017 - Tested unsuccessfully
avatar joomdonation
joomdonation - comment - 24 Mar 2017

I have tested this item ? unsuccessfully on c2af37b

There are SQL error causes by this PR:

  1. I created two text fields

  2. I created to articles

  3. I created a menu item to link to category blog layout menu option

Access to that menu item, see this error

1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') AND item_id = '2'' at line 3

The actual SQL command is

SELECT field_id,value FROM jos_fields_values WHERE field_id IN () AND item_id = '2'

Look like in some condition, field_id doesn't have any data for some reasons


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/14558.

avatar joomdonation
joomdonation - comment - 24 Mar 2017

BTW, in my test, look like the custom fields are not being displayed for Category Blog menu option. So in that case, could we stop querying the database to get data? (I haven't tested this feature much so far, so maybe I just mis-understand how it works)

avatar mbabker
mbabker - comment - 24 Mar 2017

I don't think I would hardcode the plugin context to exclude category views. In theory someone could do a category layout using fields if desired.

avatar joomdonation
joomdonation - comment - 24 Mar 2017

OK. That makes sense (although with default usage like this, it is a waste of resource to query data without using it - It might has performance impact). Also, one thing about category view which I cannot test right now because above error. Before patch, for each item, an SQL query is is generated to get data for a field, so with 2 articles, 2 fields, 4 query total. I guess if this patch works, there will only one query. Do we want to merge it to one query to get data for all custom fields of all articles (not sure if it is possible)

avatar Bakual
Bakual - comment - 24 Mar 2017

Do we want to merge it to one query to get data for all custom fields of all articles (not sure if it is possible)

Not possible with the current plugin events. You would need a new event for the whole list.

avatar laoneo laoneo - change - 25 Mar 2017
Labels Added: ?
avatar laoneo
laoneo - comment - 25 Mar 2017

@joomdonation I'v added a test if the field ids are empty. Now the error should be gone.

avatar joomdonation joomdonation - test_item - 25 Mar 2017 - Tested successfully
avatar joomdonation
joomdonation - comment - 25 Mar 2017

I have tested this item successfully on 23fac40


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/14558.

avatar joomdonation
joomdonation - comment - 25 Mar 2017

OK. It works now. Unrelated to this PR, on category blog page, still one query per article if you want to improve (in case it can be improved)

avatar laoneo
laoneo - comment - 25 Mar 2017

Open an issue for it. But as @Bakual said, it needs a new event for it, probably not a bad idea, but I'm not sure if it is doable in the 3 series.

avatar klas klas - test_item - 26 Mar 2017 - Tested successfully
avatar klas
klas - comment - 26 Mar 2017

I have tested this item successfully on 23fac40

with 100 items in list number of queries is reduced from 1000 to 100, there is also jsut a single query on single item.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/14558.

avatar klas
klas - comment - 26 Mar 2017

This is much better but still not quite there (not the fault of this PR), we have the same problem with tags.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/14558.

avatar laoneo
laoneo - comment - 27 Mar 2017

Guess RTC?

avatar rdeutz rdeutz - change - 27 Mar 2017
Status Pending Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2017-03-27 07:35:53
Closed_By rdeutz
avatar rdeutz rdeutz - close - 27 Mar 2017
avatar rdeutz rdeutz - merge - 27 Mar 2017
avatar sbouey
sbouey - comment - 27 Mar 2017

work now with Falang

avatar PhilETaylor
PhilETaylor - comment - 27 Mar 2017

Please all test again - I think this broke something in the current staging after this was merged

#14932

Add a Comment

Login with GitHub to post a comment