User tests: Successful: Unsuccessful:
The Nested table class has been the foundation for the database-stored tree structures in Joomla since version 1.6 and I've encountered several issues since then, which resulted in additional issues later on. There are reports of broken URLs because the menu table or category table got corrupted or broken permissions because the usergroup table or asset table got corrupted. Besides that, ordering often enough got mangled as well and the solution so far was to run the Rebuild task in the different components, which called Nested::rebuild(). While that method did solve a lot of issues, there are a few cases where the method fails and might even make it worse:
lft column in the database and then rebuilds it. If the process runs into a timeout (for example because the dataset is so large) this can leave the table in a worse state than before.This PR creates an alternative where the changes are done iterative. Instead of writing each row each time, it first checks the integrity and only modifies those which are broken. This can still mean a large change when there is a hole in the tree, but especially in the normal case, where the table is fine, the rebuild() process would finish a lot quicker.
The benefit is also, that the lft and rgt values are not cleared first and thus the table does not get worse than it was. Most importantly however, they process can recover gracefully when the process aborts somewhere in the middle. When 20k rows are broken and the current process can only fix 5k of those in this run, it will only have to fix the 15k broken rows left from last time. Especially for very large sites, this would allow the process to fix the whole tree eventually.
This proposal here also checks if the table has certain columns like level, alias, path and ordering and adaptively adds the features if present, unlike the current code which requires some of these fields, which is why the Assets table class needed its own version of rebuild().
Important part!!
In terms of performance, this new approach runs a lot faster. While a tree with 20k corrupted rows might take ~20s in the original implementation, this proposal would take ~5s in worst case. However, at the same time the memory consumption rises significantly. Where the recursive approach takes less than 600k of memory regardless of table size, the iterative approach takes about 1k per row it has to process. A table with 150k rows might in worst case be entirely corrupted and thus result in a memory requirement of 150 Megabytes to rebuild it. Then again, in most cases rebuild() would only touch a few rows and most likely run in under 1s. This is something that we should take into consideration.
This class comes with a set of unit- and integration-tests which I separately handed in as PR in #48227 in order to see that they pass or not on the current class and would still pass on the new class. #48227 can be closed when everyone is happy that the tests test what they should do and then we would see that they work correctly on this new implementation.
I think this would also be fully backwards compatible. At least I can't come up with a situation where this would generate issues.
The code and analysis has been written with the help of AI, but reviewed by me and thus should adhere to the AI policy we have.
Nested::rebuild()These instructions test Joomla\CMS\Table\Nested::rebuild(), the method that rebuilds the nested set
columns (lft, rgt, level, path) of a tree table from its parent_id links.
rebuild() is not called directly by a user. It sits behind the Rebuild toolbar button in three
places, behind every save and reorder of a nested item, and behind component installation. The tests
below drive it through all of those.
Use a test site and take a database dump first.
rebuild() writes to #__menu, #__categories and #__tags. If it goes wrong on #__menu you can
lock yourself out of the administrator, because the admin menu is built from that table. Several tests
below deliberately corrupt a tree, so a restore point is not optional.
mysqldump -u USER -p DATABASE > before-test.sql
Replace #__ with your real table prefix (for example jos_) in every SQL snippet.
core.adminYou will run this set of queries repeatedly. Save them somewhere. Substitute the table under test for
#__categories — the same six queries work for #__menu and #__tags.
1. Exactly one root, numbered from zero
SELECT id, parent_id, lft, rgt, level, path FROM `#__categories` WHERE parent_id = 0;Expected: exactly one row, with lft = 0, level = 0 and an empty path.
2. The numbering is dense
SELECT COUNT(*) AS nodes, MIN(lft) AS min_lft, MAX(rgt) AS max_rgt FROM `#__categories`;Expected: min_lft = 0 and max_rgt = 2 * nodes - 1. A tree of 30 rows must end at 59.
3. Every node has lft < rgt
SELECT id, title, lft, rgt FROM `#__categories` WHERE lft >= rgt;Expected: no rows.
4. No number is used twice
SELECT v, COUNT(*) AS uses FROM (
SELECT lft AS v FROM `#__categories`
UNION ALL
SELECT rgt AS v FROM `#__categories`
) x GROUP BY v HAVING COUNT(*) > 1;Expected: no rows.
5. Every child sits strictly inside its parent, one level below it
SELECT c.id, c.title, c.lft, c.rgt, c.level, p.lft AS p_lft, p.rgt AS p_rgt, p.level AS p_level
FROM `#__categories` c
INNER JOIN `#__categories` p ON c.parent_id = p.id
WHERE c.lft <= p.lft OR c.rgt >= p.rgt OR c.level <> p.level + 1;Expected: no rows.
6. The path is the parent path plus the alias
SELECT c.id, c.title, c.path, p.path AS parent_path, c.alias
FROM `#__categories` c
INNER JOIN `#__categories` p ON c.parent_id = p.id
WHERE c.path <> CASE WHEN p.path = '' THEN c.alias ELSE CONCAT(p.path, '/', c.alias) END;Expected: no rows. On PostgreSQL use || instead of CONCAT().
7. No orphans (informational — rebuild() does not repair these)
SELECT c.id, c.title, c.parent_id
FROM `#__categories` c
LEFT JOIN `#__categories` p ON c.parent_id = p.id
WHERE c.parent_id <> 0 AND p.id IS NULL;Expected on a healthy site: no rows. If there are rows, note them — they are invisible to
rebuild() and will still be wrong afterwards. That is existing behaviour, not a regression.
#__categories and note the results. This is your baseline.Expected result: every check still passes and the category list looks exactly as it did before —
same order, same indentation, same nesting.
Note: the Rebuild button rebuilds the whole
#__categoriestable, not just the extension you
are currently filtering on. Categories belonging to Contacts, Banners and News Feeds are renumbered
too. Check those list views as well.
This is the highest-risk table. Do it after Test 1, and keep your dump handy.
#__menu.Expected result: checks pass, both menus render, no item has moved between menus.
#__menuholds every menu in one tree — the site menus, the administrator menu and the hidden
"Menu_Item_Root". A rebuild renumbers all of them together. Verify at least two different menutypes.
Sport → Football → Rules, plus a second top-level tag Culture.#__tags.Expected result: checks pass, tree unchanged.
Saving an item calls rebuild() on that item's subtree only, with an explicit start node.
#__categories.Expected result: the changed category and every descendant now carry the new alias in their
path. A grandchild that was sport/football/rules becomes athletics/football/rules if you renamed
sport to athletics. No other branch changed.
Expected result: all checks pass, and the whole moved subtree appears under the new parent in the
list view with the correct indentation.
Repeat both steps for a menu item and for a tag.
Expected result: checks pass, the category keeps its children, and the new order survives a page
reload.
Repeat for Menus → All Menu Items.
This is what rebuild() exists for. Only parent_id is trusted; everything else must be recomputed.
UPDATE `#__categories` SET lft = 0, rgt = 0, level = 0, path = '';Expected result: every check passes again and the list view is restored to the correct tree. The
path column is rebuilt from the aliases, so it must match check 6 exactly.
UPDATE `#__categories` SET lft = lft + 1000, rgt = rgt + 1000 WHERE level > 1;Expected result: identical to step 4.
Installing an extension inserts administrator menu items and rebuilds #__menu.
#__menu.Expected result: checks pass both times, the sidebar is correct both times, and no leftover menu
row is orphaned (check 7).
The indexer rebuilds the taxonomy tree.
#__finder_taxonomy.Expected result: checks pass, branches show their nodes.
Only worth doing if the PR claims a performance or memory change.
INSERT INTO `#__categories`
(parent_id, lft, rgt, level, path, extension, title, alias, note, description,
published, checked_out, checked_out_time, access, params, metadesc, metakey, metadata,
created_user_id, created_time, modified_user_id, modified_time, hits, language, version)
WITH RECURSIVE seq AS (SELECT 1 AS n UNION ALL SELECT n + 1 FROM seq WHERE n < 2000)
SELECT 1, 0, 0, 0, '', 'com_content', CONCAT('Perf test ', n), CONCAT('perf-test-', n), '', '',
1, NULL, NULL, 1, '{}', '', '', '{}', 42, NOW(), 42, NOW(), 0, '*', 1
FROM seq;DELETE FROM `#__categories` WHERE alias LIKE 'perf-test-%';Expected result: the rebuild completes without a PHP memory or execution-time error, all checks
pass, and after the cleanup rebuild the numbering is dense again (check 2).
If you want to record numbers for the PR, set error_reporting to show fatals and watch for
Allowed memory size ... exhausted and Maximum execution time — those are the two failure modes
worth reporting, together with your memory_limit, max_execution_time and the node count.
Please state:
A "before" and "after" screenshot of the Categories or Menu Items list view is helpful when the tree
looks wrong, since a broken lft/rgt usually shows up as wrong indentation or a wrong order rather
than as an error message.
Please select:
Documentation link for guide.joomla.org:
No documentation changes for guide.joomla.org needed
Pull Request link for manual.joomla.org:
No documentation changes for manual.joomla.org needed
| Status | New | ⇒ | Pending |
| Category | ⇒ | Libraries Unit Tests |
Please change all @since 6.2.0 to use __DEPLOY_VERSION__
| Labels |
Added:
Unit/System Tests
PR-6.2-dev
|
||
Please change all
@since 6.2.0to use DEPLOY_VERSION