What are we actually aiming to fix or bypass here? Because I still don't see a root cause issue, just applying an arbitrary restriction based on "we know these update paths work".

Even with the update server changes applied, we have NOT identified the core issue as of yet and there should continue to be an open issue to identify the root cause of the issue preventing clean updates to the current release so that the root issue may be addressed with an appropriate fix, not just forgotten about because "m'eh, we changed the update server, good enough".

What are we actually aiming to fix or bypass here?

Long story short. When the token did not match you are redirected to the finaliseconfirm page. But in case of 3.7.0 or above the database update scripts are not run yet. So the client_id is not in the database. => error on the finaliseconfirm page

Then only code whise fix would be a try catch at all places where the client_id error can come from and fallback to the old behavior.

Or a complete rewite how com_joomlaupdate works.

for the test we need to manualy install the last version for the "Joomla update component" (3.6.1) or update it only on Joomla 3.6.0 where the auto update is enabled for the "Joomla update component" ?

for the test we need to manualy install the last version for the "Joomla update component" (3.6.1) or update it only on Joomla 3.6.0 where the auto update is enabled for the "Joomla update component" ?

No there is no need to use the com_joomlaupdate update at any time since 3.6.3 where we implemented the finaliseconfirm fallback.

What do you thing about this my change 8c6f272#diff-a296558abf749881f99f0297bb5c725bR43

maybe replace this line by a.* fix update?

maybe replace this line by a.* fix update?

hmm Isn't that code already merged?

Also it looks like this method is not triggered anywhere? https://github.com/joomla/joomla-cms/search?utf8=%E2%9C%93&q=getMenus&type= Or I'm missing anything?

with J3.5.1

In Joomla update component, entered custom update URL, and enabled usage of it

• Was proposed update to J3.6.5, worked
• then was proposed update to J3.8.0, worked

Long story short. When the token did not match you are redirected to the finaliseconfirm page. But in case of 3.7.0 or above the database update scripts are not run yet. So the client_id is not in the database. => error on the finaliseconfirm page

Aware of that. But...

• The client_id schema change was part of 3.7 by way of 4e156fa
• To the best of my knowledge, the widespread reports of the client_id error did not start until 3.8's release

This tells me there is a code change between 3.7 and 3.8 regarding whether the client_id column was used in a query and inherently preventing a page from rendering correctly while the application is in an unstable state.

All of this is why #18127 is open to begin with. Because our post update routine is flawed. Period.

If we are aware of the specific code change that is causing the issue, that can help us in a lot of ways going forward, including:

• Heightened awareness of making changes in specific parts of the system that may be used in the middle of the update routine
• Being able to defensively code around these parts of the system to prevent failures due to the system being in an unstable state
• Being less reliant on a "fix" of requiring additional update steps

If we don't understand what is the cause of these issues and we just roll fixes without said understanding, we aren't doing anyone any justice, and we just end up continuing to add things like

More or less I try to guess.
Column #__menu_types.client_id does not exists before 3.7.5.
It was added at:

com_admin/sql/updates/mysql/3.7.0-2016-11-19.sql:1:ALTER TABLE `#__menu_types` ADD COLUMN `client_id` int(11) NOT NULL DEFAULT 0;

On 3.6.4? 5? there is a task com_joomlaupdate.update function that start using token to check security.

If there is token then relogin is not needed and module menu is not called.
If token is missing, J3.5.1 older then relogin is required and we have our problem.

ok on Joomla 3.5.1.

On Joomla 3.3.0 error: Notice: Undefined property: JUpdate::$downloadurl in /administrator/components/com_joomlaupdate/views/default/tmpl/default.php on line 60

Notice: Trying to get property of non-object in /administrator/components/com_joomlaupdate/views/default/tmpl/default.php on line 60

and can't update, can't see the download url.
PHP 5.4.25

3.6.1 is when the CSRF check was added.
3.7.0 is when the #__menu_types.client_id column was added.
3.8.0 is when update failures regarding the column not being present became more prevalent.

By this logic, an upload and update style update should fail if coming from any release before 3.7.0. If I've traced history correct, #9612 added the feature to 3.6.0. Assuming the column were added at 3.8.0, then an upload and update style update should fail on any release before 3.8.0.

If I'm remembering things correctly, the client_id error is happening more frequently with any 3.6.5 or earlier update going directly to 3.8.0. It is not occurring with 3.7 updates to 3.8 (or if it is I haven't seen it as frequently).

If I've gotten things correct in this comment (and I could be misremembering, I'll be the first to say it) the update server changes only help people who are doing updates the "normal" way which requires no manual package download/upload, we aren't fixing things for upload and update.

on 3.6.0 the update point to 3.8.0 and with no update the "Joomla update component" to 3.6.1 error
Unknown column 'a.client_id' in 'where clause'

I don't know if the stack trace helps

<b>Fatal error</b>:  Uncaught JDatabaseExceptionExecuting: Unknown column 'a.client_id' in 'where clause' in libraries\joomla\database\driver\mysqli.php:650

Stack trace:
#0 libraries\joomla\database\driver.php(1691):
JDatabaseDriverMysqli->execute()

#1 libraries\src\Menu\MenuHelper.php(314):
JDatabaseDriver->loadObjectList()

#2 libraries\src\Menu\MenuHelper.php(337):
Joomla\CMS\Menu\MenuHelper::loadXml(Object(SimpleXMLElement), Array, Array)

#3 libraries\src\Menu\MenuHelper.php(133):
Joomla\CMS\Menu\MenuHelper::loadXml(Object(SimpleXMLElement), Array)

#4 administrator\modules\mod_menu\menu.php(89):
Joomla\CMS\Menu\MenuHelper::loadPreset('joomla')

#5 administrator\modules\mod_menu\mod_menu.php(26):
JAdminCssMenu->load(Object(Joomla\Registry\Registry), true)

#6 libraries\src\Helper\ModuleHelper.php(201):
include(in libraries\joomla\database\driver\mysqli.php</b> on line 650

It may come from changes in #16451

It may become after changes in #16451

yes
the query is loaded from the XML file: (about line 208)
administrator\components\com_menus\presets\joomla.xml

(also similar but not used in our case, since joomla.xml is the default on upgrade, is)
administrator\components\com_menus\presets\modern.xml

@mbabker If you ok with cause find by @csthomas and @ggppdk then I will test this PR.

Foe me this fix the problem:

diff --git a/libraries/src/Menu/MenuHelper.php b/libraries/src/Menu/MenuHelper.php
index 2737bc29f3..ebc0ed6351 100644
--- a/libraries/src/Menu/MenuHelper.php
+++ b/libraries/src/Menu/MenuHelper.php
@@ -311,7 +311,15 @@ class MenuHelper
                                        $query->innerJoin($iJoin);
                                }
 
-                               $results = $db->setQuery($query)->loadObjectList();
+                               try
+                               {
+                                       $results = $db->setQuery($query)->loadObjectList();
+                               }
+                               catch (\JDatabaseExceptionExecuting $e)
+                               {
+                                       // The database has not yet been updated
+                                       $results = array();
+                               }
 
                                // Skip the entire group if no items to iterate over.
                                if ($results)

How to test:

install J3.6.0 and try to update to 3.8.0. After error add this pach and refresh page to relogin

@mbabker

The client_id schema change was part of 3.7 by way of 4e156fa
To the best of my knowledge, the widespread reports of the client_id error did not start until 3.8's release

This tells me there is a code change between 3.7 and 3.8 regarding whether the client_id column was used in a query and inherently preventing a page from rendering correctly while the application is in an unstable state.

3.3.6 can (with difficulty) be updated to 3.7.5 #18012 (comment) but trying to update 3.3.6 to 3.8 gives a database error and blank screen. This would tend to support your statement.

Set to "closed" on behalf of @franz-wohlkoenig by The JTracker Application at issues.joomla.org/joomla-cms/18192

closed as having Pull Request #18197

so please what is the solution
what to do to fix i can't get into the admin panel
update from 3.5 to 3.8

Apply my fix and complete installation.

You can also download whole file from https://github.com/csthomas/joomla-cms/blob/e0fb8b8da3dafc19cf52e1458e9463243cd91b96/libraries/src/Menu/MenuHelper.php
and replace it.

Then go to administrator/index.php?option=com_joomlaupdate&view=update&layout=finaliseconfirm
login and complete installation.
I have not tested this steps.

thank you very much it worked as magic
but in the admin panel after finished the upgrade it show me that message
Warning

JInstaller: :Install: Error SQL Duplicate entry '453' for key 'PRIMARY'
Files Update: SQL error file DB function failed with error number 1062
Duplicate entry '453' for key 'PRIMARY'
SQL =
INSERT INTO #__extensions (extension_id, name, type, element, folder, client_id, enabled, access, protected, manifest_cache, params, custom_data, system_data, checked_out, checked_out_time, ordering, state) VALUES
(453, 'plg_editors-xtd_module', 'plugin', 'module', 'editors-xtd', 0, 1, 1, 0, '', '', '', '', 0, '0000-00-00 00:00:00', 0, 0);
Joomla Version Update Status
Your site has been updated. Your Joomla version is now 3.8.0.

and also the front end work but without css why i do not know

The sql error come from

grep -B2 -rnIG "plg_editors-xtd_module"
3.5.0-2015-10-13.sql-1-INSERT INTO `#__extensions` (`extension_id`, `name`, `type`, `element`, `folder`, `client_id`, `enabled`, `access`, `protected`, `manifest_cache`, `params`, `custom_data`, `system_data`, `checked_out`, `checked_out_time`, `ordering`, `state`) VALUES
3.5.0-2015-10-13.sql:2:(453, 'plg_editors-xtd_module', 'plugin', 'module', 'editors-xtd', 0, 1, 1, 0, '', '', '', '', 0, '0000-00-00 00:00:00', 0, 0);

IMO before you start installation to 3.8 you have some inconsistency in your database.
You probably did manually update files to 3.5 or above and then used discover to install above plugin.

@csthomas

Apply my fix and complete installation.

I did that, works fine, thanks a lot!

also thank you very much it worked as magic

but still anther problem
the front end of the site is without css
http://www.dralkhadhari.com/index.php/en/

@baraaabdelhady this is not related. Open an Thread for that on the forums where we will, help you. This is not a support platform

@gwsdesk
yes you right but i said it because that happened after the upgrade
the site was working OK unit i upgrade so i think may be something happen that need to be fixed
and thank you for your replay

@baraaabdelhady Update to 3.8.0 is more resource intensive. Please post on forums an run FPA. i will respond after work.

sorry for the late join

If we roll these update server fixes, I can live with that as it helps users run into issues less often, but I'm not OK with us not identifying and making the best effort we can at fixing the real issue because there is a way to get around it that works.

the updated server fix doesn't mean we stop to investigate the real issue

@baraaabdelhady

but i said it because that happened after the upgrade
the site was working OK unit i upgrade so i think may be something happen that need to be fixed

If you can reproduce the issue updating from a fresh install of a Joomla version then post it as an issue with exact steps to replicate the error. Otherwise post in https://forum.joomla.org/ for help in what you have done wrong/failed to do correctly. Thanks

@baraaabdelhady once again your issue might have been caused by the upgrade but still this Github site is not the place to solve individual user issues. You will have to post on the Joomla forums. The Issue Tracker (Github) is to solve bugs and not to help individual users so post on the forums (PLEASE!)

Hello, so in an update from 3.7.5 to 3.8.1 it crashed.

The error was "unable to write htaccess.txt" (which does not exist because it has been renamed .htaccess).

The admin backend is now inaccessible with the error:

Error displaying the error page: Class 'Joomla\CMS\Menu\Tree' not found: Class 'Joomla\CMS\Menu\Tree' not found

The file "MenuHelper.php" does not exist in the file tree:

[root@vn html]# find -name 'MenuHelper.php'
[root@vn html]#

Making updating it impractical.

A request to administrator/index.php?option=com_joomlaupdate&view=update&layout=finaliseconfirm does display the login dialog.

Perhaps of note is that this is a fresh site (effectively very close to a clean install)

@gp2u As I wrote to @baraaabdelhady this update is nead more resources. Please post values of:
memory_limit
max_execution_time

@wojsmol can you please stop giving support here? This is not the forum. These sort of questions need to be posted and answered on the forums!

@gwsdesk Now I give only very limited "support" - #18192 (comment) I wrote to help separate update failure from Joomla! bug

@gp2u Please open topic in https://forum.joomla.org/viewforum.php?f=710

In case anyone is visiting this thread, searching for the solution

Having turned on error logging my error was Error displaying the error page: Class 'Joomla\CMS\Menu\Tree' not found: Class 'Joomla\CMS\Menu\Tree' not found

As noted above I could not find the MenuHelper.php module grep-ing the source tree. This was because the entire libraries/src folder was missing.

I did the following on the command line

cd /var/www
mkdir tmp
cd tmp
wget https://downloads.joomla.org/cms/joomla3/3-8-1/Joomla_3.8.1-Stable-Full_Package.zip?format=zip
unzip Joomla_3.8.1-Stable-Full_Package.zip?format=zip
cd ../html/
rm -rf libraries/
cp -r ../tmp/libraries/ .
chown -R apache:apache libraries/

After which it worked. Note that first I tried just copying libraries/src over but this failed. Then I edited libraries/src/Menu/MenuHelper.php with the suggested patch but that failed too. Replacing the libraries directory as a block resolved my specific issue.

Now for the Joomla maintenance people here I would like to mention a consistent issue (assumption error) with Joomla updates. If you work on the command line, and as root rather than apache then files within the Joomla tree can end up owned by root and unwritable. The crash in my case did not relate specifically to that but rather to the udpate widget not being able to handle the fact it could not write a file (htaccess.txt because this had been renamed). Had it existed it would have been owned by apache and been writable, but as it did not exist it was trying to write html/htaccess.txt where html was owned by root. This only impacts the writing of files in the root html/ dir.

Anyway it seems to me that it would be sensible to perform a few checks, with error catching, before initiating an update. That would include making sure write permission is available. It would also seem sensible to have a "safeWrite" function that was used for all file writes, so in the event a file can not be written this could be a) caught and logged and b) that file could be written to a temp dir like the html/tmp dir with the same file path that the file should have been put at.

Judging by the state of the Joomla file system and the missing libraries/src it is apparent a rm -rf type command has been executed before trying to write files which inevitably means the system will be broken in the event of a write failure. To make that more stable and atomic writing to dir.new first, then renaming dir to dir.old and then renaming dir.new and then deleting dir.old would be a hell of a lot more atomic.

Judging by the number of threads on the forum reporting the 3.8 update breaking sites there is a substantial issue with how the update process works.

Judging by the state of the Joomla file system and the missing libraries/src it is apparent a rm -rf type command has been executed before trying to write files which inevitably means the system will be broken in the event of a write failure.

Two "rebuttals" here.

1. The libraries/src directory is new to 3.8. So depending on where unpacking the ZIP package failed, it isn't all that surprising that it still doesn't exist.
2. The update order of operations for the filesystem is extracting the ZIP package in place then removing files in a specified list (which is only reliably available after extraction because we need that list to come from the new release). No rm -rf style operation is attempted in this though, at most you might get a rm -r operation.

To make that more stable and atomic writing to dir.new first, then renaming dir to dir.old and then renaming dir.new and then deleting dir.old would be a hell of a lot more atomic.

Not an option unfortunately. The current update path allows users to update a site without taking it offline, depending on the site's traffic and what the update entails there may be a small number of requests which have a 500 response in the middle of that otherwise there is no requirement to put a site offline to run the update. Even if an atomic process were possible, it would call for a major rewrite of the update system to use even more system resources (because you would basically have to fully extract the ZIP package to a temporary directory then move the contents of the temporary directory into the production paths) and this would probably raise the number of reported errors.

Judging by the number of threads on the forum reporting the 3.8 update breaking sites there is a substantial issue with how the update process works.

The root issue in the update system has already been explained in other issues. And that is trying to use the Joomla application itself to update the Joomla installation. Because the finalization process is a normal request to the Joomla application (handled the same as going to an edit page in com_content at a high level), the system is unstable during that request and when you start adding in extensions which overload the core APIs it just makes everything too unpredictable. Fix that particular design issue and we're fine.

The fix posted by @csthomas worked like charm!

Thanks, man. You saved my life :)