@Hackwar please can you confirm ?

I can confirm this behavior. Just tested and get similar result. Just questioning if this is just related to "legacy option" enabled? "curious"

Have you tested adding a hidden menu item of type List All Categories with Root as top level?

no, don't work also with menu item of type List All Categories

This is expected behavior. You are expecting a random menu item to link to some other random content. You will have to create menu items fitting to your sites structure to have these URLs work.

Yes.
I hope we can hide the new router options in Joomla 3.7 and release the 3.7.0 stable.
In Joomla 4 we can re-insert the new router options.
But Joomla 3.7 can't be release with the new router options visible. It is not work.

But Joomla 3.7 can't be release with the new router options visible. It is not work.

That's a pretty bold claim to make. Because the new system does not have 100% behavior parity with the existing one it does not work? It has been said from the day this work all started that the new router was going to have a different behavior and workflow than the existing one, it would be a pretty big slap in the face at this point to hide it just because it doesn't match the existing system (honestly I'd just rip out the new one instead of trying to make it work like the existing one).

With all respect for @Hackwar huge effort for this community (for which big thanks!) I do agree with @AlexRed on the non-release for Joomla 3.7. The found issues must be addressed before being released

No menu item of type List All Categories, no 404 page, no orphan articles ecc...
For me Joomla must be better than this.

Nobody is saying the issues will not be fixed. However, the code should not be changed to adapt to the existing workflows and behaviors (some of which are quite bugged and cause other issues). The new router is in essence a clean break from that. It does require some retraining, but based on what I have read and seen of it, that changed workflow will result in better managed websites with less possibility of there being "odd" behaviors as is the case now with the router.

I have mentioned that the known (too many) issues should be fixed before we release them. I never mentioned (nor did Alessandro) that the new router should adapt the current workflows. I just believe that it is wrong to introduce such an important and much awaited new way of handling the router without addressing the already known issues. We should not release them just for the sake of the fact that we had announced them (!) for Joomla 3.7 and forfeit the issues we have discovered which are too big too ignore as mentioned by Alessandro and supported by me

The base issue here is a result of a bad configuration. You can't have a child category of a single article menu item, which is what this configuration is. You HAVE to have at least two menu items for this configuration; one for the single article and one for the category tree. If that category tree thing isn't working then yes there's a bug in that configuration, but I personally would stamp a "won't fix" over the "set up a site with a single menu item which is of type single article and all routing should work correctly for all articles and categories" issue.

If that is the issue that needs to be chased down, please update the title and description to reflect that. But as they read now, this is not something that requires a core change.

this is an old issue, the new issue (release-blocker) is: #13978

@gwsdesk Then tell me which issues you mean? There is exactly one issue that can not be resolved to bad configurations of the website and that is that the new routing throws less 404s and fails to the homepage instead of some random page. But as I said earlier already, that is not fixable until we are allowed to break backwards compatibility. Until then, people will have to know what they are doing and use a plugin like I described in #12410.

@AlexRed Not really. We are still talking about the same mis-configured websites and the same problem and same solution. Fortunately, nobody has to use the new routing, but can simply stay on the legacy routing with his misconfigured website until 4.0.

Correct and this is a release-blocker also for me

@Hackwar You cannot expect from the average beginning Joomla user that he is knowing what she/he is doing. You cannot say they can use the legacy since they would not know the difference (between what is given in current -legacy so to speak- and 'modern' and post their misfortunes on the Joomla forums where we have to explain that we have not addressed the issues before the release but that we might do that in the future......

Again I do not agree releasing something that brings a lot of problems for Joe-the-average-user (which is our market!) without addressing the issues. And @Hackwar a tremendous thanks for all you have invested...that is for sure not the issue

So the solution to people misconfiguring their websites is for Joomla to drop new, disabled-by-default features? It's not as if there is no way to write a script that discovers if there are menu items for login, registration, forgotten password, missing menu items for content, etc....

and before that next comment is coming: No, we can not do those checks in the router. The router needs to be fast, lean and small. And I don't see it as my job to write a setup tool for Joomla, too, besides the category system, the router, a rewrite of JForm, the password system, the breadcrumb code that most extensions out there use and the several other features that I'm working on for J4.0 or have been working on in the past.

So the solution to people misconfiguring their websites is for Joomla to drop new, disabled-by-default features?

For sure not but the problem we will face with non-corrected found issues (the bigger ones as this is such bigger one) is that nobody knows the difference unless we create wiki entries in 'docs' to explain the difference and consequences of both options and the non-working ones etc which is absolutely not only already difficult for non end-users to understand let alone for the average Joomla user to understand (not going to 'newbies' even) So instead increasing the problems we should avoid them by just postponing this release till we can test possible changes that address this. We should we be vote against a proper and clean release for major changes as (in this case) the (fabulous in principle) new router is?

@Hackwar If we require a plugin to throw 404's. then we need to ship that by default in core i think. we can't have this many redirects as default (even if we then subsequently remove the plugin as throwaway code in j4)

as I said, that plugin would then again have to be disabled by default, since otherwise it would break many sites. See what I did in 4.0-dev in the application router in the parse() method at the end. If a component router does not delete segments, (which is not the case in way to many routers) the URL would be legit, but still throw a 404 because of that.

Now, we've been discussing all of this for at least a year now. Either we accept that we fucked up in the past, need to make it better in 4.0 and live with the consequences for now or someone else than me needs to look into this whole matter, find a good solution and prove me wrong. But so far everybody has just complained and bitched and moaned and not actually provided a different (working) solution. In the end, this code has been available like this for the last 3 years and the problem has been around since 1.5.

Agree with

need to make it better in 4.0
And we did not complain.....we gave very relevant reasons (users in mind!) to postpone it indeed to J4.0

@gwsdesk you are missing the point that this is not a new issue with the new routing, but has been around since 1.5 and is just slightly worse with the new routing than before.

I mean I can't find a way to trigger a 404. Which is more than slightly worse :/

@Hackwar I am not missing any point and I do understand all the past flaws you point at very much correctly but that is not the issue I and with me many others describe here. We have very severe issues in the new solution that need to be addressed before releasing them and releasing them now is giving the Community a new piece of code that is from the beginning flawed again if we do not address this properly before launch of this great (as meant to be) improvement

You're looking for the past flaws to be reintroduced. That's not a good thing. The 404 thing needs a patch, we agree on that. But making sure you get "pretty" URLs in a obviously not configured correctly website should be off the table, that's where a lot of the black hat magic in the current router lives (and add in Itemid and you're in routing/display hell).

I agree with Michael. As long as we solve the 404 thing I'm happy with this to ship as is.

No I am not looking at past flaws to be re-introduced. That is a (very wrong) incorrect observation to say the least. We do need to patch known flaws before releasing and that is what I am advocating

Its about Communication: Whats about new router, who benefit, ... Marketing has to be easy to understand who should use and who should wait on 4.0.

@franz-wohlkoenig it is not.... It is about serving the community with proper code/solutions benefiting all imho

With all respect for everyone i agree with @AlexRed
As i've already said in the past, the new router project must be considered a failure imho. It has not real advantages, it's more complex and as of now i don't see it working in a stable and usable fashion. The only advantage for the end user is only one: remove the numeric ID from urls, that's a thing already present in the 'legacy' router that could have been activated since years without the new router challenge.

@joeforjoomla And as I've said in the past, you have no idea what you are talking about.

Just for shits and giggles, here are the benefits of the new router:

• URLs will get the right Itemid, regardless of them being sent through some random class (e.g. ContentHelperRoute, ContactHelperRoute, UsersHelperRoute, etc.)
• URLs will get an updated Itemid, even when they are part of an article content
• Users can have different URL styles if they want (and the third party community picks up on this)
• Developers now have a universally working masterplan for their components router, creating high quality routers instead of the crap that you see in 90% of the cases.
• Developers can write their router with easy to create boilerplate code, again making sure that the quality of component routers are improved.
• Fixing a bug in the routers can now happen at one point in code and not in {amount of installed components on site}, which also means that extensions benefit from those fixes without releasing something new themselfs.
• Router code has been unittested and largely cleaned up.

Downsides:

• Requires more properly configured sites
• Throws less 404s than before, which can not be fixed in a B/C way

You can say none of this about the current router. I don't call that a failure.

The new router must be valued in the situation, when updating a 3.6.5 site to 3.7 and then enabling the new router. This should be result in a better situation than before. If we can't achieve this that the project "new router" failed for 3.7. IIRC it was promised that the implementation of the new router will be in a B/C manner, saying that problems can be only fixed in 4.0 fulfils not the promise made.
Furthermore I have serious doubts that we can go into a update to 4.0 with the statement that people have to update the URL/Menu structure before and telling them "in the mind of 4.0 this is a misconfiguration".

The main problem I see at the moment is the 404 problem, we can't start with a new router and then make it worse as it was before. Soft 404 are a real problem in terms of SEO much more as having an id within the url. If that can't be fixed than I will go a step further as I did and remove the new router from the 3.7 release.

I did spend some time at the weekend to look at the router and especially at the 404 problem (used a default joomla install + url = /lkasjdflkajsdlf). The problem comes form a change in the routing process. If in RouterSite@parseSefRoute the system can't find a matching menu item it sets the active menu to the default menu item. That's ok because this is how it was in the past. The difference come later. When it comes to the component router then old system tries to match the url to a known element and when this doesn't worked (for com_content in this case) it sets view=article and id=0 what is not valid and we end up in a 404. The new router checks if the url is valid for the component and when not (because it used the active menu item, set to default before) it'll use the default menu item. Tata!

So the wrong thinking here is that the component router doesn't come into play, so the component router can't say this isn't a valid url I can't route to a valid view.

It is also a problem within the new code and not the old code so I think it can be done in a B/C manner. Would be good if some else can have a look at it.

The interesting part is in:

libraries/cms/router/site.php, Method parseSefRoute
libraries/cms/component/router/rules/standard.php, Method parse

So that are my findings so far, comments are welcome.

Please hide the new router options in Joomla 3.7.
No menu item of type List All Categories, no 404 page, no orphan articles ecc...
The marketing team are using the "New router" feature like a fantastic one and the dev team hiding behind a "Experimental Routing"... it is not correct, it is unethical.

This will be addressed within the Production Department.

seems already done today: 0c31f1d
hiding behind a "Experimental Routing"
And the marketing show the video in the social: https://www.youtube.com/watch?v=NtwDE5Paxd0
and the images about the fantastic new router

As I said, this will be addressed, because the current handling is nothing short of a project wide failure.

@Alex I already told you many times that the marketing stuff is NOT a problem.
don't use this as an argument please...
Deal with the code, marketers will deal with marketing

For me is a problem. In the Joomla.org news https://developer.joomla.org/news/671-joomla-3-7-beta-3-released-for-testing.html is the image about the fantastic new router.
For me is better wait the RC to public the marketing materials and ask to translator to work in the translations. But I probably I'm wrong.

yes.. for you!

ok, sorry for my wrong opinion. I humbly apologize to everyone and I close the discussion.

For me is better wait the RC to public the marketing materials and ask to translator to work in the translations. But I probably I'm wrong.

That is too late to begin preparing or publicizing material for a release. Feature promotion and teasers should go out well in advance. Of course that's going to assume a well oiled machine is running between at least Production and Marketing. By the time the release actually comes out, for the community members who have been seeing that stuff for months it should really be old news at that point. Take a look at Symfony's blog for an example of what I consider to be efficient marketing for an upcoming release; features start getting teased/blogged about within hours sometimes of being merged.

@rdeutz The difference between the old and the new router is, that the new router has a defined behavior and the old router has not. The old router is like C. It works somehow, but there are a lot of undefined behaviors and they behave in one way in one situation and in another in another situation. What will be a misconfigured site with the new router, already is a misconfigured site in the current router.

The problem of the component router not being able to throw a 404 is, that you can not know if the parts that you failed to parse are maybe parts that will be matched by another rule and/or if they will be matched in the component- or application-router. Since the behavior of the application- and component router in the past was not defined, the routers were not forced to remove the parts that they matched and thus we have no indication if all parts of a URL were successfully matched or not. Since we can not change behavior in 3.x, we have to stick with this until 4.0.

Now, the code has been out there for literally years. No one cared to review this and discuss this with me. Now that we are close to release, people start noticing what I've been saying for equally many years and freak out. I'm trying to fix this part of Joomla for eight-fucking-years and looking at this right now, it seems as if everybody again is whining for this to be removed again. At the same time it doesn't seem as if the project really is interested in finally moving towards 4.0 to clean up not only this, but a shitload of other stuff and that wants me to rage and vandalise something. People are complaining that Joomla doesn't move forward, but when it does, they are complaining that it moved at all. Fuck. This. Shit. Get your act together and grow some balls or stop the project and lets all move over to Wordpress. ARG! drops mic

@AlexRed "For me is better wait the RC to public the marketing materials and ask to translator to work in the translations. But I probably I'm wrong." Sound more like being convenient than wrong to do the work when others done the testing to make it work. :)
One of the finest extra tasks of localising/translating the software is to report back findings to be fixed at an early stage as possible to ensure it works across all languages as supposed.

Marketeers just do their thing. So should translators. Often and early for best possible outcome :)

+1 for @rdeutz and @AlexRed
+1 for people who chosen to rename as 'experimental'
Because the new router is nothing better than i failure IMHO
To not forget: https://www.indiegogo.com/projects/advance-the-joomla-url-router#/

Yes, never forget that campaign from me. If anyone ever thinks about doing something for the Joomla project and financing that with crowdfunding: Don't. Simply don't. People will hate you and mistreat you for years, regardless of the outcome, and blame you for the failures of the Joomla project. At the same time you will never "earn" more than $1/hour, since even with the most carefull planing, you will still waste months in discussing and defending the code that you are trying to bring into the project. Rather sell your soul to ebay or Google and then go berserk on the project and be treated like a hero, while others can clean up the mess that you caused. But never ever again attempt crowdfunding.

I wanted to ask. Why it is not possible to raise an error in the component router, if a page does not exist? I think, if you manage to do that, it will outweigh all existing disadvantages.

Both the application- and the component-router are basically a pipeline of matching functions, which are applied one after another on the input and which transform this to an output. So you have for example the matching function to discover the &format= of the request by looking at the suffix. Is it .html? Is it .json? It should then remove that part that it matched from the input. When all the input is gone, we have successfully parsed and matched all parts of the URL. If something is left, we did not match all input successfully and should throw a 404. However, both the component router and the application router can be extended by additional rules or matching functions, which for example take /blablabla/page-1 and translate that page-1 to &start=0 for our pagination. We can not know what those functions are doing and we can not know if they did their part successfully. Unfortunately the old router did not require to remove parsed input, so that there are tons of components out there, which routers successfully parsed the URL, but did not remove the input. So we don't know if they did their part successfully and the remaining input will be parsed by some matching function in the application router or if we already failed.

Logically, can we add a control function that would run at the very end of pipe, comparing the URL that is retrieved from the input to the request uri? I know, it is out of scope this issue, just telling you my thoughts.

@Hackwar you are telling us now for years that this router problems can be fixed in a B/C manner and now weeks before the release you are saying it can't be done in a B/C manner. It's easy to forget this so take it as a reminder.

It is risible when we release a "modern" router and have to say "oh it can't figure out if a url is valid but it is really great and we are moving forward".

Our definition of backwards compatible: The site has to still work like before after the update.

That is the case. That is why we have the switch between modern and legacy routing.

The "Modern" router is as "Modern" as this flying machine

is to

Once you understood that Category ROOT is not to be used as top level but that one has to create a new root category (with all other categories as children) and create systematically a hidden (or not) List All Categories menu item with THAT new root category as Top level for each component (and also for each language on a multilingual site), then almost all is fine, except for the 404s which behave even weirder than "legacy".

Changing the name "Modern" to "Experimental" is a good move imho. A bit late it seems but better than nothing.

It is indeed frustrating, but there will be many

before we reach

Once you understood that Category ROOT is not to be used as top level but that one has to create a new root category (with all other categories as children) and create systematically a hidden (or not) List All Categories menu item with THAT new root category as Top level for each component (and also for each language on a multilingual site), then almost all is fine

But isn't that fundamentally the architecture of joomla?

Wll, we did not have to do it before. ROOT was a real ROOT.

I would not even use the word "experimental' I would rather use "technical preview" with a footnote that people can have a look and test what is coming in Joomla 4.0 (if this all can indeed be resolved)

Once you understood that Category ROOT is not to be used as top level but that one has to create a new root category (with all other categories as children) and create systematically a hidden (or not) List All Categories menu item with THAT new root category as Top level for each component (and also for each language on a multilingual site), then almost all is fine

This is going to be a disaster for existing sites and I cannot believe that we will be able to sell this to our user base

@infograf768 Did I say that the root behavior was right that way? No, I did not. I simply asked you to report that issue in another place, instead of putting it into a totaly unrelated issue. I have in fact added a fix for that in #14322.
And if you keep on insulting me, don't complain if I do the same.

@gwsdesk So, then tell me what is broken now that worked perfectly in the old router.

This is going to be a disaster for existing sites and I cannot believe that we will be able to sell this to our user base

Did you see that @Hackwar

@gwsdesk Did you read my previous comment? My question still stands.

Guys this conversation has gone way beyond the line of civility. I'm locking this conversation