No Code Attached Yet
avatar jasonscottmontoya
jasonscottmontoya
1 May 2017

Steps to reproduce the issue

Upgraded manually to Joomla 3.7
Discovered upgrades and updated database
Cleared cache and made all Joomla components active
Was replicated on three different joomla upgrade websites

Expected result

Admin menu for components should not have changed

Actual result

Components menu made certain items like contacts and joomla upgrade disappear. Inconsistent menu items disappeared across the three websites - If I manually go to the url of the missing components, they still work, but they're just gone in the admin menu.

System information (as much as possible)

PHP Built On Linux res192.servconfig.com 2.6.32-673.26.1.lve1.4.20.el6.x86_64 #1 SMP Tue Dec 27 17:42:53 EST 2016 x86_64
Database Version 5.6.33-log
Database Collation latin1_swedish_ci
Database Connection Collation utf8mb4_general_ci
PHP Version 7.1.3
Web Server Apache
WebServer to PHP Interface cgi-fcgi
Joomla! Version Joomla! 3.7.0 Stable [ Amani ] 25-April-2017 15:36 GMT
Joomla! Platform Version Joomla Platform 13.1.0 Stable [ Curiosity ] 24-Apr-2013 00:00 GMT
User Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36

Additional comments

I can provide login information as needed.

Votes

# of Users Experiencing Issue
1/1
Average Importance Score
3.00

avatar jasonscottmontoya jasonscottmontoya - open - 1 May 2017
avatar joomla-cms-bot joomla-cms-bot - labeled - 1 May 2017
avatar wojsmol
wojsmol - comment - 1 May 2017

@jasonscottmontoya Check value of menutype column in #__menu table for componentss which disappeared. This column should have value of main


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/15719.

avatar jasonscottmontoya
jasonscottmontoya - comment - 1 May 2017

Ok, I've gone in and found the discrepancy - Correcting it per your direction worked. Any idea how this occurred in the first place?

avatar wojsmol
wojsmol - comment - 1 May 2017

For same reson this query whas not run https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_admin/sql/updates/mysql/3.7.0-2017-01-17.sql. If you don't use menu named main or menu in frontentd run this query on phpmyadmin. Replace #__ with table prefix.

avatar jasonscottmontoya
jasonscottmontoya - comment - 1 May 2017

ok, thanks.

On Mon, May 1, 2017 at 12:24 PM, Wojciech Smoliński <
notifications@github.com> wrote:

For sam reson this query whas not run https://github.com/joomla/
joomla-cms/blob/staging/administrator/components/com_
admin/sql/updates/mysql/3.7.0-2017-01-17.sql. If you don't use menu named
main or menu in frontentd run this query on phpmyadmin.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#15719 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aa8_qPqnp3vOU2qZDS5VbG28_laZrOTwks5r1gcxgaJpZM4NNFMa
.

--
Jason Montoya

Path Of The Freelancer Is Now Available On Amazon
https://www.amazon.com/Path-Freelancer-Actionable-Flourish-Freelancing/dp/1540735419/

avatar franz-wohlkoenig franz-wohlkoenig - change - 2 May 2017
Status New Discussion
avatar bolando
bolando - comment - 2 May 2017

I had the same issue: my Joomla 3.6.5 menu was named "main" and after upgrade to 3.7 its items went to admin.

avatar wojsmol
wojsmol - comment - 2 May 2017

@bolando When more or less this website was created?

avatar bolando
bolando - comment - 2 May 2017

The site was created a few years ago in Joomla 2.5 and upgraded to Joomla 3.6 a few months ago.

avatar wojsmol
wojsmol - comment - 2 May 2017

@bolando Have you used a quick start package for a template?

avatar bolando
bolando - comment - 3 May 2017

No

avatar schultz-it-solutions
schultz-it-solutions - comment - 3 May 2017

Same issue here on multiple - like two dozen - sites (updating from 3.6.5 to 3.7.0). Submitting the named SQL query solved the issue, but where does this issue come from in the first place???

avatar shanenefdt
shanenefdt - comment - 4 May 2017

I have also has exactly the same problem - updated from 3.6.5 to 3.7.0. This query addressed the problem successfully.
There is however also another issue since the upgrade - that of the Send Test Email from the Server Tab under Global Configuration - keep getting:
Error
An error has occurred while fetching the JSON data: HTTP 500 status code. Internal Server Error
Wondered if this was perhaps related?

avatar isherwood
isherwood - comment - 17 May 2017

In my case the main menu was being merged into the Components admin menu.

Based on the advice above I gave my main menu a different type string ('top') before attempting the upgrade a second time. This time the menus were not corrupted.

avatar jasonscottmontoya
jasonscottmontoya - comment - 17 May 2017

ok, thanks for sharing.

On Wed, May 17, 2017 at 12:19 PM, Clint Buhs notifications@github.com
wrote:

Based on the advice above I gave my main menu a different type string
('top') before attempting the upgrade a second time. This time the menus
were not corrupted.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#15719 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aa8_qEhCpBL1LBTH6xGVhbhWRs_34bBLks5r6x4CgaJpZM4NNFMa
.

--
Jason Montoya - 770-265-1933
Path Of The Freelancer Is Now Available On Amazon

http://amzn.to/2p1nDd2 http://amzn.to/2p1nDd2

avatar inspry
inspry - comment - 19 May 2017

Thanks for sharing. That sql update snippet brought the admin back all the menu items.

avatar affectit
affectit - comment - 26 May 2017

@wojsmol thanks for sharing. worked perfectly. I used a quickstart package though and not upgraded Joomla

avatar tonypartridge
tonypartridge - comment - 31 May 2017

@shanenefdt that's an error unrelated to this. Search git please.

avatar richard67
richard67 - comment - 7 Jun 2017

Please test PR #16577 as soon as I have updated the testing instructions there. Btw. does anybody know which Joomla version can be used to create a menu type "main" or "menu" for the site (frontend)? Does that work on a 3.6.5? Would be good to know for easy testing instructions.

avatar richard67
richard67 - comment - 7 Jun 2017

PR #16577 is ready. Please test.

avatar richard67
richard67 - comment - 18 Jun 2017

Guys, please test #16577 (issue tracker link see here), otherwise the issue might not be solved with 3.7.3, and other people will run into the same trouble with menus being messed up when updating a version lower than 3.7.0 to 3.7.3.

avatar Quy
Quy - comment - 22 Jun 2017

Close per fixed in #16577

avatar joomla-cms-bot joomla-cms-bot - change - 22 Jun 2017
Status Discussion Closed
Closed_Date 0000-00-00 00:00:00 2017-06-22 03:48:47
Closed_By joomla-cms-bot
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2017
avatar franz-wohlkoenig franz-wohlkoenig - change - 22 Jun 2017
Closed_By joomla-cms-bot franz-wohlkoenig
avatar joomla-cms-bot
joomla-cms-bot - comment - 22 Jun 2017
avatar franz-wohlkoenig
franz-wohlkoenig - comment - 22 Jun 2017

closed as having PR #16577.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/15719.

avatar franz-wohlkoenig
franz-wohlkoenig - comment - 22 Jun 2017

Thanks for Hint, @Quy

avatar BastianWie
BastianWie - comment - 4 Jul 2017

Was that solved in 3.7.3? I just upgraded multiple sites (25 in total) from 3.7.2 to 3.7.3 and found out that some entries in the menu where missing. I then check #__menu and I still can see that the menutype for com_banners, com_messages, com_newsfeeds_categories and a lot others are still on "menu" rather then on "main".

The sites I updated where created with different Joomla versions in the past and then upgraded to the latest versions over time (always with uploading the files via FTP and replacing the existing ones, then running the DB upgrade button in the admin GUI).

Running the following:

UPDATE <myDBName>.<myJoomlaDBsuffix>_menu SET menutype = 'main' WHERE menutype = 'menu';

fixed the issue for me

avatar richard67
richard67 - comment - 4 Jul 2017

@BastianWie It was already solved in 3.7.0, but that has caused issues for others who had user-defined menus for the frontend. So in 3.7.3 the update statement in the schema update sql for 3.7.0 was corrected. But there was no change made for updating a 3.7.2 to 3.7.3. If in past the update from a version before 3.7.0 to version 3.7.0 has not fixed the problem for some reason, then updating to 3.7.3 will not change anything.

If you say you had a 3.7.2 with menu items of menu type "menu" for the backend (admin), then it means your update to 3.7.0 has not been performed correctly.

The method you use for updating is since a while not officially supported anymore.

The Joomla! update component offers now (beside the live update) the possibility to upload the update package, and so there is really no need anymore to do it in the old (unsupported) way.

See the instructions on how to update in the Joomla! docs.

When you update with the old method, DML (data manipulation language, i.e. insert, update, delete) statements in schema update scripts will not be run, because the db fix only cares about the database schema, i.e. about DDL (data definition language, e.g. create table and so on).

And so then you update in that way, the fix from 3.7.0 for the menu type names is not performed.

When you update a pre-3.7.0 with the Joomla! update component, the complete schema update scripts will be run once if necessary, and so also the fix from 3.7.0 (which was corrected in 3.7.3) will be run, and all will be ok.

A site which has been updated in past from version to version only with the copy or ftp the files and then run the db fix method will very likely be the source of many issues for these reasons (dml of schema updates did not run).

I am not sure if someone (or me) should make a PR here to fix such sites.

Since the Joomla Update Component exists, the copy or ftp and so on method was only an emergency solution in case Joomla Update component cannot be used, it never was the preferable way to update.

So or so it seems to me that the 3.7.0 was not clean somehow.

avatar richard67
richard67 - comment - 4 Jul 2017

I think if peopple refuse to follow the updating instructions which are easy to find in the docs and linked in every release announcement, it is not a bug of Joomla, it is a bug of the people. Sorry to say.

avatar Hackwar
Hackwar - comment - 8 Jul 2017

Just FYI: I ran into the same issue with a site which was updated over the last few years with with the joomla update component. I also did not have a menu with name "main".

avatar wojsmol
wojsmol - comment - 8 Jul 2017

@Hackwar Validate the data in the #__menu table in a copy of the database that was made before the update.

avatar richard67
richard67 - comment - 9 Jul 2017

@Hackwar The point is not which method has been used in past. The point is that people still use the "copy or unzip the files and then run db fixer" method to update to 3.7.3 and then complain that the fix from #16577 has not been applied.

avatar richard67
richard67 - comment - 9 Jul 2017

Well, one thing is that it was fixed in PR #16577 for the update from pre-3.7.0 versions to 3.7.0, not for updating a 3.7.0 or 3.7.1 or 3.7.2 to 3.7.3.

The reason for that was that I just wanted to fix the wrong schema update script for 3.7.0, assuming that those who already did the update had to fix the problem manually anyway after the update and so have solved their problem, or had to fall back to before the update, which would make the PR #16577 to have an effect (if using the update component) when the do again the update of the pre-3.7.0. to 3.7.3.

And so I assumed that there should not be a version 3.7.0 or 3.7.1 or 3.7.2 having that problem.

If this assumption is wrong, it may make sense to add an update script for 3.7.4 which also includes the fix, so when updating to 3.7.4 using the update component would fix it.

To see if this is necessary it would help if someobody who still has that problem on a 3.7.1 or 3.7.2 or 3.7.3 would provide information here about the content of the "#__menu" and the "#__menu_types" table (columns menutype, title, client_id and in case of "#__menu" also the id should be sufficient).

avatar richard67
richard67 - comment - 9 Jul 2017

Guys, do you think I should make a PR which repeats the fix from #16577 in new update script 3.7.4-2017-xx-yy.sql so versions 3.7.0 to 3.7.3 will be fixed. too? I have already prepared that, but am waiting for feedback. But also this would not work if not using the Joomla Update component to do the update to a 3.7.4 (or patched 3.7.4-dev for testing).

avatar mbabker
mbabker - comment - 9 Jul 2017

We really need to stop using JSchemaChange* as a means for ensuring system integrity. SQL statements that need to be replayable and do not change the database schema need to be moved to a point where they can be triggered in PHP (and inherently reused if needed).

So no. Do not add an update delta to replay that change. If this is such a critical change that it has to be verifyable and something that can be retriggered through that magic black box we call the database fix button (which does more than validate the database if you ever trace its internals), it needs to be handled right and we need to stop monkey patching the system with crap solutions.

avatar richard67
richard67 - comment - 9 Jul 2017

@mbabker Then maybe a fix in script.php (which is also called by the database fixer) like it is done already for other stuff could be a way?

avatar richard67
richard67 - comment - 9 Jul 2017

I am not sure how critical it is, I never had that issue. But disappearing menus seem critical to me.

avatar mbabker
mbabker - comment - 9 Jul 2017

In my eyes, the only queries that should be in those changeset files should be queries related to altering the database schema. Queries that are changing data records do not belong there. A database migrations system (for core Joomla that's JSchema*, outside Joomla you have tools like Phinx) is designed for managing schema changes and should only be used for data changes when a schema change necessitates it.

avatar richard67
richard67 - comment - 9 Jul 2017

Sur, I already understood that and agree. But now we have it different, and my question was then if it is worth to be done in script.php. The statements of #16577 do not do any harm if multiply applied and the conditions in the statements are not met after the 1st apply.

avatar BastianWie
BastianWie - comment - 9 Jul 2017

By the way, couldn´t we use the "Post-Install Messages" to add a news here that the old upgrade way to overwrite files via FTP is no longer supported and shouldn´t be used? At some time it could become critically and I personally think that there are much more persons who never have seen that. The "Post-Install Messages" already informed about the two factor authentication, so why not for the correct way to upgrade a installation? I think informing the users during the next upgrade via "Post-Install Messages" wouldn´t harm and might help to spreed out this important news.

avatar brianteeman
brianteeman - comment - 9 Jul 2017

Not much point it being in the post install message - its too late then

http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
Virus-free.
www.avg.com
http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On 9 July 2017 at 20:52, Bastian W. notifications@github.com wrote:

By the way, couldn´t we use the "Post-Install Messages" to add a news here
that the old upgrade way to overwrite files via FTP is no longer supported
and shouldn´t be used? At some time it could become critically and I
personally think that there are much more persons who never have seen that.
The "Post-Install Messages" already informed about the two factor
authentication, so why not for the correct way to upgrade a installation? I
think informing the users during the next upgrade via "Post-Install
Messages" wouldn´t harm and might help to spreed out this important news.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#15719 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPH8ctmSeEScUiFO1YVWOJHlflkmKccks5sMS91gaJpZM4NNFMa
.

--
Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
https://brian.teeman.net/ http://brian.teeman.net/

avatar mbabker
mbabker - comment - 9 Jul 2017

You can still do an upgrade through FTP if you know what you're doing. It's not that FTP upgrades are not supported, but more often than not, users aren't performing all of the required steps to do an upgrade. It was fine in Joomla 1.5 days because the upgrade really way nothing more than "replace these files", but since 1.7 upgrades have supported numerous extra steps under the hood as well (database updates and clearing the system cache being two important tasks).

If you're upgrading outside of using the upgrade component and you basically aren't following the steps outlined in https://gist.github.com/mbabker/d7bfb4e1e2fbc6b7815a733607f89281 (not saying you have to use that script but it does trigger all the things), your upgrade can't be considered complete.

avatar wojsmol
wojsmol - comment - 9 Jul 2017

@mbabker Does this script also execute UPDATE queries from sql files? I remember cases from the Polish Joomla forum where most probably did not do it

avatar mbabker
mbabker - comment - 9 Jul 2017

My script uses the update component's model to emulate everything that happens after a normal update has unpacked the update package's contents onto your site. That code calls into the JSchema API, and that is the API that will not run data statements (INSERT, UPDATE, etc.) after the first time a file is parsed.

So if the data queries are somewhere in our admin script.php file then they can always be checked and executed in that magic black box fix routine (and inherently anything that uses it). If the data queries are only in SQL files, they can only be processed once and cannot be validated again.

avatar wojsmol
wojsmol - comment - 9 Jul 2017

Would it be possible to rewrite this script to emulate upload and install. Omitting the zip file download phase? It should temporarily terminate https://docs.joomla.org/J3.x:Update_from_3.7.x_to_3.7.1_Timeout and any problem, updating to 3.8.x, 3.9.0 and 4.0 resulting from the same entity.

avatar mbabker
mbabker - comment - 9 Jul 2017

It basically does emulate upload and install. The difference is you manually unpack the ZIP archive on your server then trigger the script whereas upload and install unpacks the package the same way the "normal" update routine does.

avatar janeblonde
janeblonde - comment - 19 Jul 2017

Suffering badly from this issue.

I have several joomla sites all 100% up to date i.e using 3.7.3 (I didn't notice missing anything during updates as those components are rarely used).

I've tried all fixes in:
https://issues.joomla.org/tracker/joomla-cms/15719

and:
#16852

but I always run into an error like #1054 - Unknown column 'oldmain' in 'where clause' or #1054 - Unknown column 'client_id' in 'where clause'

I've reinstalled via joomla update and ftp but nothing works.

When I look at the Menus > Menu Items > Administrator I see the missing items all set to Menu but the Select Menu is empty.

In the databasse I can see all items again in #__menu (see: http://i.share.pho.to/9d28ee92_o.png), and when I look in the #_menu_types I only see front end menus (see:http://i.share.pho.to/9d28ee92_o.png), should I see one for Menu or a column for asset_id?

Any help would be greatly appreciated, going out of my mind here!


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/15719.

avatar wojsmol
wojsmol - comment - 19 Jul 2017

@janeblonde

Unknown column 'client_id' in 'where clause'

Run this query in phpmyadmin

ALTER TABLE `#__menu_types` ADD COLUMN `client_id` int(11) NOT NULL DEFAULT 0;

Replace #__ with dbprefix

avatar janeblonde
janeblonde - comment - 19 Jul 2017

@wojsmol
Thank you, I've ran that and then also the below query which for the first time has been successful however there is no change in the admin menu, I've also rebuild the menu and cleared caches, but still no change. So some progress but still no cigar:(

Here's how the tables now look:
http://i.share.pho.to/66c367ce_o.png
http://i.share.pho.to/9c154962_o.png

UPDATE `e8w6u_menu`
   SET `menutype` = 'main_is_reserved_133C585'
 WHERE `client_id` = 0
   AND `menutype` = 'main'
   AND (SELECT COUNT(`id`) FROM `e8w6u_menu_types` WHERE `client_id` = 0 AND `menutype` = 'main') > 0;

UPDATE `e8w6u_modules`
   SET `params` = REPLACE(`params`,'"menutype":"main"','"menutype":"main_is_reserved_133C585"')
 WHERE `client_id` = 0
   AND (SELECT COUNT(`id`) FROM `e8w6u_menu_types` WHERE `client_id` = 0 AND `menutype` = 'main') > 0;

UPDATE `e8w6u_menu_types`
   SET `menutype` = 'main_is_reserved_133C585'
 WHERE `client_id` = 0 
   AND `menutype` = 'main';

UPDATE `e8w6u_menu`
   SET `client_id` = 1
 WHERE `menutype` = 'main';

UPDATE `e8w6u_menu`
   SET `menutype` = 'main'
 WHERE `client_id` = 1 
   AND `menutype` = 'menu';

UPDATE `e8w6u_menu`
   SET `menutype` = 'main',
       `client_id` = 1
 WHERE `menutype` = 'menu'
   AND (SELECT COUNT(`id`) FROM `e8w6u_menu_types` WHERE `client_id` = 0 AND `menutype` = 'menu') = 0;

DELETE FROM `e8w6u_menu_types`
 WHERE `client_id` = 1
AND `menutype` IN ('main', 'menu');
avatar wojsmol
wojsmol - comment - 19 Jul 2017

@janeblonde You use custom admin menu named Admin Menu?

avatar janeblonde
janeblonde - comment - 19 Jul 2017

@wojsmol
I have a front end menu called adminmenu (title:Admin Menu), this contains some links for front end editing, do you think this could be the issue?

avatar wojsmol
wojsmol - comment - 19 Jul 2017

@janeblonde No, I needed more information about this menu.See #15719 (comment)

avatar janeblonde
janeblonde - comment - 25 Jul 2017

@wojsmol
I don't have a custom admin menu, it's the system default for admin, it looks like this:
http://i.share.pho.to/f563e442_o.png
Note: if you click Select Menu there is nothing to select. If I create a custom admin menu then this list disappears.

avatar wojsmol
wojsmol - comment - 25 Jul 2017
avatar janeblonde
janeblonde - comment - 25 Jul 2017

@wojsmol
Do you think it could be a missing menu?
All missing menu items in #__menu have main (see:http://i.share.pho.to/b9c3ceb6_o.png)
but in the #__menu_type table there is no corresponding menu for main (nor menu).

avatar wojsmol
wojsmol - comment - 25 Jul 2017

@janeblonde See this comment https://github.com/joomla/joomla-cms/blob/2d341103cdd6b6fe652c17776e63fb9f6a17dd7d/administrator/components/com_admin/sql/updates/mysql/3.7.0-2017-01-17.sql#L52-L55 So there is something else is wrong in #__menu table, see this issue #15938 - if nested tree is broken then menu mey disapered.

avatar janeblonde
janeblonde - comment - 25 Jul 2017

@wojsmol
I've ran that query, I don't see that it's made a change (see: http://i.share.pho.to/8f47a304_o.png & http://i.share.pho.to/34fcc001_o.png). There is no change on the admin, they all look to be set to Menu and Published (see: http://i.share.pho.to/aefd50a6_o.png).

re #15938 I don't see an error like:

Failed publishing 1 menu item as at least one of its parents is unpublished or one of its children is checked out.

But what is interesting is the mention of Banners, this is not in my list of published Admin menu items and it is included as a menu item (see:http://i.share.pho.to/6db3f6a2_o.png) but if I compare them in the #__menu table they seem more or less the same.

avatar janeblonde
janeblonde - comment - 25 Jul 2017

systeminfo-2017-07-25T13_50_00+00_00.txt for reference I've added the system info file here, the highlights would be:
Linux cloud.jai.ie 3.10.0-229.14.1.el7.centos.plus.x86_64
Database Version | 5.5.52-MariaDB
PHP Version | 7.0.21

avatar wojsmol
wojsmol - comment - 25 Jul 2017

@janeblonde Plese attach sql dump of #__menu if possible.

avatar janeblonde
janeblonde - comment - 25 Jul 2017

certainly...
it contains #__menu_types and #__menu.
dev2.zip

avatar wojsmol
wojsmol - comment - 25 Jul 2017

@mbabker Plese try download attached zip - I cen't - for me this is github issue whitch shoud be reported.
@janeblonde Plese upload this zip to your server temporally.

avatar janeblonde
janeblonde - comment - 25 Jul 2017

Add a Comment

Login with GitHub to post a comment