? ? Pending

User tests: Successful: Unsuccessful:

avatar ggppdk
ggppdk
10 Sep 2018

Pull Request for Issue # .

Summary of Changes

Category manager creates 1 query per category to get the assigned item counts by state
These are used to displays the columns
total published, total unpublished, total archived, total trashed

Testing Instructions

  1. Open 2 Tabs of Category manager and 2 Tabs of Tag manager
  2. Filter the 2 Tabs of Tag manager by "Tag Type" article (click search tools button)
  3. Apply patch
  4. Refresh 2nd TAB of Category manager and 2nd Tab of Tag manager

Both 1st and 2nd tabs should show the same counted items per state


Category managers to test

  • articles, banners, contact, newsfeeds, and for fieldgroups management

Tags manager:

  • repeat test for every listed Tag Type, in the Tag Type Filter (click search tools)

Expected result

1 query to get the total assigned items for the categories shown by current page

Actual result

20, 50, 100 queries to get the assigned items for the categories

Documentation Changes Required

None

avatar ggppdk ggppdk - open - 10 Sep 2018
avatar ggppdk ggppdk - change - 10 Sep 2018
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 10 Sep 2018
Category Administration com_content
avatar ggppdk ggppdk - change - 11 Sep 2018
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - change - 11 Sep 2018
Category Administration com_content Administration com_banners com_contact com_content com_fields com_newsfeeds Libraries
avatar ggppdk
ggppdk - comment - 11 Sep 2018

even better can be done on some high level "category" CLASS/HELPER

yes that was my thought too
that is why those local variables in the first commit for the columns names, that seemed redundant ...

I intended to replace countItems() method everywhere, but i see that countTagItems() has the same problem of 20, 50, 100 queries instead of 1

so i have added a countRelations() method to parent helper class that replace both methods,
actually the methods are not replaced instead they call the parent class method like this
parent::countRelations($items, ...,$config);

avatar ggppdk ggppdk - change - 11 Sep 2018
Title
[Performance] Category manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1
[Performance] Category manager / Tag Manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1
avatar ggppdk ggppdk - edited - 11 Sep 2018
avatar alikon
alikon - comment - 11 Sep 2018

countTagItems() you can accuse me for that 😨

avatar ggppdk ggppdk - change - 11 Sep 2018
The description was changed
avatar ggppdk ggppdk - edited - 11 Sep 2018
avatar HLeithner
HLeithner - comment - 11 Sep 2018

We have a PR for such a function for j4 #21667 I will rework it to do the same optimization as this PR. It's really need I ask my self why I didn't do this.

Atm I wait for #20693 to get merged to get the direction how to solve the component naming issue.

avatar ggppdk
ggppdk - comment - 11 Sep 2018

That PR has a completely different purpose than PR #21667,
PR #21667 tries to make a reusable method in a trait

Furthermore this PR not only replaces multiple queries for category assignments with 1 query (this is the purpose of this PR)
but this PR also fixes same issue with Tag counting which PR #21667 does not even touch

A note
the code here is agnostic in regards to which state / condition exist
thus it can be rather easily ported to J4

avatar ggppdk
ggppdk - comment - 11 Sep 2018

Of course if you would want to do same optimization inside PR
#21667 for J4 then please do, as i lack time to do more work now
thanks

avatar alikon
alikon - comment - 11 Sep 2018

please, can I ask you if can we concentrate our effort on 3.x series for a while !!
if we can improve performance in the current staging....it will'be a real gain
4.x ..... it will come later......

avatar ggppdk
ggppdk - comment - 11 Sep 2018

please, can I ask you if can we concentrate our effort on 3.x series for a while !!
if we can improve performance in the current staging....it will'be a real gain
4.x ..... it will come later......

sure, just i mention that the configuration object of the method countRelations() is structured having in mind to make easy

  • to be used by J3 components with custom states / custom states subsets
  • to be used (after ported) with a custom subset of "conditions" for J4 too
avatar alikon
alikon - comment - 11 Sep 2018

sorry for my idiosyncrasy... i just want to plant my foots on earth and not on mars

avatar laoneo
laoneo - comment - 12 Sep 2018

20, 50, 100 queries to get the assigned items for the categories

This function is used only in the back end category list view. As far as I can see it, it is executed only once. But please prove that I'm wrong.

avatar ggppdk
ggppdk - comment - 12 Sep 2018

This function is used only in the back end category list view. As far as I can see it, it is executed only once. But please prove that I'm wrong.

Sure here is the proof
please notice the existing code inside
helper classes
e.g. for categories of articles ($items contains category objects)

		foreach ($items as $item)
		{
			$item->count_trashed = 0;
			$item->count_archived = 0;
			$item->count_unpublished = 0;
			$item->count_published = 0;
			$query = $db->getQuery(true);
			$query->select('state, count(*) AS count')
				->from($db->qn('#__content'))
				->where('catid = ' . (int) $item->id)
				->group('state');
			$db->setQuery($query);
			$articles = $db->loadObjectList();
		...
		}

and then also i tested to confirm that they are executed, and then spent considerable time for this PR also revising code for tags assignments too

You can confirm the multiple queries by looking at the Debug queries slider,

and then also inside the items loop after:
$articles = $db->loadObjectList();

you can add

echo 'Executed : ' . $query . '<br><br>';
avatar ggppdk
ggppdk - comment - 12 Sep 2018

I speak of the multiple sql queries (20, 50, 100) that a single call to the methods will do

i never said that the method is called multiple times

avatar laoneo
laoneo - comment - 12 Sep 2018

Now I got you.

avatar laoneo
laoneo - comment - 12 Sep 2018

What's the idea behind the config variable?

avatar HLeithner
HLeithner - comment - 12 Sep 2018

ggppdk

Furthermore this PR not only replaces multiple queries for category assignments with 1 query (this is the purpose of this PR)
but this PR also fixes same issue with Tag counting which PR #21667 does not even touch

I didn't noticed that you changed the complete PR to more then changing the DB query sorry I missed this I talked about this part.

avatar ggppdk
ggppdk - comment - 12 Sep 2018

What's the idea behind the config variable?

basically it makes possible the method to be reusable
we now have 1 method in the parent class and 5 countItems() and 3 counTagItems() methods in descendants helpers classes call it

also copying this from my previous answer

sure, just i mention that the configuration object of the method countRelations() is structured having in mind to make easy

  • to be used by J3 components with custom states / custom states subsets
  • to be used (after ported) with a custom subset of "conditions" for J4 too

e.g. banner has different table name and different column name than content

also the counting for fields in fieldgroups does not use 'catid' column, the column name is 'group_id'

avatar alikon
alikon - comment - 16 Sep 2018

I have tested this item ✅ successfully on 0028cc8


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

avatar alikon
alikon - comment - 16 Sep 2018

I have tested this item ✅ successfully on 0028cc8


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

avatar alikon alikon - test_item - 16 Sep 2018 - Tested successfully
avatar Quy
Quy - comment - 27 Sep 2018

I have tested this item ✅ successfully on 3d7db8c


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

avatar Quy Quy - test_item - 27 Sep 2018 - Tested successfully
avatar Quy Quy - change - 27 Sep 2018
Status Pending Ready to Commit
avatar Quy
Quy - comment - 27 Sep 2018

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

avatar csthomas
csthomas - comment - 29 Sep 2018

@ggppdk Can you explain why you have:

	public static function countItems(&$items, $config = null)
	{
		$config = $config ?: (object) array(
			'related_tbl'   => 'banners',
			'state_col'     => 'state',
			'group_col'     => 'catid',
			'relation_type' => 'category_or_group',
		);
		return parent::countRelations($items, $config);
	}

IMO this would be better:

	public static function countItems(&$items)
	{
		$config = (object) array(
			'related_tbl'   => 'banners',
			'state_col'     => 'state',
			'group_col'     => 'catid',
			'relation_type' => 'category_or_group',
		);
		return parent::countRelations($items, $config);
	}

If someone would like to change some parameter then should use BannersHelper::countRelations($items, $config) directly.

avatar ggppdk
ggppdk - comment - 29 Sep 2018

@csthomas

indeed if someone would want to count relations with a custom table / custom case,
then someone could call somenameHelper::countRelations($items, $config)

which is the method inherited by the parent class

So indeed it seems to me that it can be as you say,
we do not need $config = null variable in methods

public static function countItems(&$items, $config = null)
public static function countTagItems(&$items, $extension, $config = null)

they can be (have same signature like before this)

public static function countItems(&$items)
public static function countTagItems(&$items)
avatar ggppdk
ggppdk - comment - 30 Sep 2018

Please remove RTC

in last CS commit the table aliases are no longer added correctly (i think they are not, i will test)
plus i will remove the unused / redundant parameter as suggested by @csthomas

avatar franz-wohlkoenig franz-wohlkoenig - change - 30 Sep 2018
Status Ready to Commit Pending
avatar franz-wohlkoenig
franz-wohlkoenig - comment - 30 Sep 2018

Status back on Pending as stated above.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

c7928a4 12 Oct 2018 avatar ggppdk CS
23a580e 12 Oct 2018 avatar ggppdk CS
avatar ggppdk
ggppdk - comment - 12 Oct 2018

Ok
the table aliases were correct
so just rebased, removed unused $config variable for methods (as suggested by @csthomas),
and make a CS fix required by appveyor

i retested, everything seems corrects,
but this needs again testing

avatar csthomas csthomas - test_item - 22 Oct 2018 - Tested successfully
avatar csthomas csthomas - test_item - 22 Oct 2018 - Tested successfully
avatar csthomas
csthomas - comment - 22 Oct 2018

I have tested this item ✅ successfully on 3a580e4ba0f8ba273cf85fa63cfeda8ab8c30a8.

There is a problem with mark this test as success on github.
I got error "Bad credentials" at
https://issues.joomla.org/tracker/joomla-cms/22117

avatar csthomas
csthomas - comment - 22 Oct 2018

I have tested this item successfully but I got error "Bad credentials" at https://issues.joomla.org/tracker/joomla-cms/22117

avatar Quy
Quy - comment - 24 Oct 2018

I have tested this item ✅ successfully on 23a580e


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

avatar Quy
Quy - comment - 24 Oct 2018

I have tested this item ✅ successfully on 23a580e


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

avatar Quy Quy - test_item - 24 Oct 2018 - Tested successfully
avatar Quy Quy - change - 24 Oct 2018
Status Pending Ready to Commit
avatar Quy
Quy - comment - 24 Oct 2018

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

avatar mbabker mbabker - change - 1 Nov 2018
Status Ready to Commit Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2018-11-01 02:34:25
Closed_By mbabker
Labels Added: ?
avatar mbabker mbabker - close - 1 Nov 2018
avatar mbabker mbabker - merge - 1 Nov 2018

Add a Comment

Login with GitHub to post a comment