14 errors in unit tests and some minor codestyle issues.

Sorry, the majority of those errors was a typo, a missing +. Now everything should be fine.

BTW: The PHP 5.6 tests throw a notice before the test is started. Something is wrong there... (Unrelated to this PR)

Hello Hannes, thanks for delivering this!

I've yet to test it and even to look at the code, but just reading your description, and especially your frustration for having to drag behind the weight of the B/C anchor, made me think something:

Wouldn't it be possible to introduce in com_config "experimental" switches for testing new (possibly B/C breaking) upcoming technologies? Those switches should be accompanied by humongous warnings stating that they are meant only for testing purposes, that most 3rd party extensions will not work when they are turned on and that there is no firm commitment that the functionality will be implemented in upcoming official releases.

This will allow 3rd party extensions developer to prepare for "the future" and core developer to receive an early feedback...

Just my 2 (probably O.T.) cents...

Hi @smanzi, that would indeed be a possibility, but I would strongly advise against it. There are 4 reasons against this:

1. Joomla already has too many options and actually needs to cut down on those. Users will be confused for sure with further options. They are already unable to cope with the FTP settings and we have way to many people that have that set, even though they don't need it and they don't have an idea why they did it.
2. At the same time, it means that we have to maintain 2 separate code blocks and have to find ways to elegantly wove this into our system. Should the new system have the same class names as the old one? Which one gets put into the default position and which solution overrides the other system? You will have such discussions for each and every change, in addition to our current discussions.
3. Third party developers will have a nightmare to support all this. Considering the routing, a third party developer might say that he wants to use the new routing system, since it is far less code to write and its easier to understand. That means a user that wants to use that extension, needs to enable the new routing system. Now if for some reason, the old and new system are incompatible and the user wants to use an extension that only supports the old routing, he is screwed. That means that every third party developer has to cater for both situations, old and new, actually increasing the work instead of reducing it.
4. Our project is notoriously incapable of supporting 2 branches of a code at the same time. A good example is/was JRegistry and JParameter. JParameter was one of the corner stones of Joomla 1.5 and was made obsolete in 1.6 by JForm and using JRegistry. Only after 1.6 was released did someone notice that JParameter has not been touched for 2 years and did not work anymore.

Based on these points, I would not introduce such options as a general measure. Especially the first point is important to me. Compare Joomla and how much options Joomla provides simply when writing an article compared to Wordpress. Yes, the two systems are very different, but having several magnitudes more options than a system that aims at the same market, does not throw the best light on J.

@Hackwar To be honest I agree only with your points 2 (strongly agree) and 4 (possibly agree).

Solutions for the other points:

• the switches could be totally hidden (no UI for setting them, just tinker with configuration.php or a separate "experimental.php" config file)
• In no way 3rd party developers should deliver officially supported (by them, I mean) code that assumes any of such experimental switches is turned on.

On the other hand I concur that smoothly integrating two "lines of code execution" in the core can be quite a mess and will most likely imply altering the "current line of code".

I see this as something that should be reserved for exceptional cases, like probably a "new router" is, not something to be used for trivial alternatives.

The router is by 80% aimed at third party developers, since it provides them a way to write a router in 5 lines of code and they don't get a serious headache just because of that aspect of their component. There would be little gained if third party devs did not adopt this code as quickly as possible.

So... how do you see the future? An "early" 4.0 (will allow breaking B/C) or the frustration of having at hand fantastic code ... that can't be used?

First of all, there are ways to get what we want right now without breaking B/C. And then I don't know what you mean with an "early" 4.0. Nothing is promised for 4.0. If the PLT decides to start work on 4.0 after 3.4 is released, so be it. 4.0 could mean that we have this router and some other minor details and that is it. Or it means something completely different.

by "early 4.0" I was meaning exactly what you said: "this router and some other minor details". Well, maybe not just minor ones: I don't have examples at hand right now, but I would say everything that we are eager to do but can't because of the B/C commitment

but... can you elaborate on the concept of "there are ways to get what we want right now without breaking B/C"?

These stages things for example are a way to rewrite the current router to a rule based concept which is backwards compatible in a lot of ways. There are also questions how far our backwards compatibility goes. If we for example said that non-SEF URLs are not covered by that backwards compatibility claim, then we could do different things while still be part of 3.x.

@Hackwar I want to test this: should I apply #5105 too?

You can test this without #5105. 5105 however needs testing, too.

got it!

@test success

• Applying this patch doesn't seems to have any negative impact with the tested component (com_content, several views)
• SEF URLs exactly as before (which is a mixed blessing, IMHO....)
• NON SEF URLs same as before

Question: should this be tested with SEF engines (like e.g. sh404) for compatibility or is this guaranteed to be B/C in this context?

Personal note: ehmm... are you aware many of those who crowdfunded your initiative are expecting you to get rid of the ArticleID from SEF URLs, aren't you? I Understand this is not acceptable in the core because of B/C, but do you plan to release a system plugin (or anything else) to achieve that?

Regarding sh404sef: Old code will be backwards compatible, but if you are using code written for whatever Joomla version this one will be released in, you might get conflicts. If sh404sef does not use the core router and instead deploys its own stuff, they would be disabling these stages effectively.

Regarding your personal note: This is a place where I plan to add an option to disable or enable certain features of the routing system, which would include the IDs. So it will be part of Joomla and it will be in in a backwards compatible way.

Perfect!

I found an older version of sh404SEF in my filesystem and quickly glanced through its code to find its router integration. From what I could tell, they're extending JRouterSite but none of the methods or vars changed in this URL are in their extended class, so this shouldn't break compatibility with their code from that aspect. @shumisha if you'd like to test this and confirm that things will continue to work well, it would be greatly appreciated.

In general with regards to backward compatibility, we can't control what happens if an extension is extending a library class and we change the internals of a method to extend what it is able to do. If the base API continues to behave the same pre- and post- change, then to me it is a B/C change. Unfortunately, once code is rewritten to make use of those newer features, that would cause an arguable B/C break if the extended class isn't updated as well. It's a risk we take any time we change the internals of a class or method to add new features.

Hi all,
Thanks Michael for the head up. I don't have time right now to get to the details of this, hopefully tomorrow. I can make the following very quick comments:

1 - strictly about sh404SEF, I do extend JRouterSite, but to be honest that's not really needed and I can probably put proxies in place if some methods I need/use were changed, which I doubt. sh404SEf is really API-compliant in that matter and only uses the additional rules it attaches to the site router.

2 - Foreseeable issues may more likely be if the routing system changes its behavior. LanguageFilter plugin is a good example. If Hannes suggestion to change it's behavior to NOT set/unset the language code were applied, that is broken between a pre-processing and a post-processing stage, then the data received by whatever routing rules are added will be different from what it is today. LanguageFilter routing has been causing much issues, and all SEF extensions (except sh404SEF) simply disable it for multilingual sites (and replicate its features elsewhere).
I have found some logic to keep it enabled and still be able to properly parse urls, but again I would expect that to break if the data provided to me is changed.

As Michael outlined, I think a B/C break would happen if changes were made to the language filter plugin routing rules to take advantage of this. Otherwise, we should be good and I'm sure I can accommodate that PR, maybe without having to do anything actually. If the LF plugin is modified in such a way however, then I think that should not happen before 4.0

I'll get back to this post in the coming days with some test results.

@Hackwar I think something that could be quite useful would be to keep and make accessible by all parsing or building rules the original input data (ie the original $uri) before it's been modified by any of the attached rules or the component itself. One reason for the difficulties with the language filter plugin is what you outlined, ie the language information is lost on the other parser/builder after the LF plugin has handled it.
So instead/on top relying on the LF to only remove the query string (when building) at the postpreocessing stage, it'd be very nice to simply store the original data in a place where any parsing rule can access it.

Hi @weeblr-dev, I've been working on the language filter plugin in PR #5140. Please look there, too.

Regarding storing the original data somewhere: I'm a bit hesitant regarding this. I personally decided against it, since I think it is not only a clean way to clean up after you, but also a way to override something. It allows me to prevent the default code from working on this stuff by removing it from the query. But if there is a majority to implement something like this, I would not fight for dear life.

Hannes, I've been testing with your PRs #5140, #5264 and #5276 in a multilingual environment and I have some results I would like to share with you hoping that you find them useful.
I'm posting the results here as this seems to be "the mother of all PRs": sorry if I'm wrong.

The test have been performed using:

2 languages:

• English en-GB en (default)
• Italiano it-IT it

3 menu structures

• Main Menu 1 item: home (Featured Articles, default home for all)
• Italiano 1 item: pagina-principale (Featured Articles, home for it)
• English 1 item: home-page (Featured Articles, home for en)

3 categories

• uncategorized (all) ID=2
• test (en) ID=8
• prova (it) ID=9

2 articles (both flagged as featured)

• foobar (en, in test, ID=1)
• casino (it, in prova, ID=2)

menu items, categories, and articles have been associated between themselves

"Remove URL Language Code" (for default language) is set to: No

Multilingual Status is:

From the above structure I had expected the following URLs to be valid and generated:

/en/
/en/8-test/1-fubar
/it/
/it/9-prova/2-casino

What I get is instead:

/en/
/en/home-page/8-test/1-fubar
/it/
/it/pagina-principale/9-prova/2-casino

i.e.: the aliases of the home pages are not stripped away

Language switcher does his duty, no problem.

The "expected URLs" do works if entered manually, but the bad news are that also "chimeric" URLs do work, like e.g.:

/it/home-page/8-test/2-casino
/en/pagina-principale/9-prova/1-fubar
/it/pagina-principale/8-test/2-casino
etc...

While others do give a 404, like:

/it/home-page/8-test/1-fubar
/en/pagina-principale/9-prova/2-casino

The discriminating factor seems to be that the language code must match the language of the item requested (disregarding categories and menu items in between)

P.S: Of course the above has been performed with the latest 'staging' at the time of this writing (head at b858931), integrated with the 3 above PRs.
I've been unable to integrate #5285 and did not tried integrating #5294 & #5295

I have the awkward and embarrassing feeling I was badly wrong about the URLs I was expecting...

The 'chimeric' URLs issue remains, as before (without this PRs), so I would say... 'business as usual', with 100% B/C

Sorry! Sergio

@smanzi I know what you mean. I actually expect the same URL and I'm pretty sure that this was the case in an earlier version of Joomla 3.x. I did not respond to you yet, because I first wanted to check the situation thoroughly. It could very well have been that my code was wrong in that regard... But this is a good example why I want to keep the current routers as a legacy option like I told @okonomiyaki3000 in #4267. I will still look into this one further and see if this is the earlier behavior or not.

@Hackwar Hannes, I was pulled to other things this week, but I just did a quick test tonite with this patch #5264 applied on a stock 3.3.6, together with sh404SEF current and I couldn't see any issue at first look. That was in a multilingual environment, but didn't apply #5140 yet. WIll do later and report any finding. Didn't test with other extensions ofc, but looks good.

#5140 looks good as well.

sprintf('The "%s" stage is not registered (%s).', $stage, __METHOD__) just to know where it comes from?

Codestyle and METHOD should be fixed.

Sorry, now it should be correct...

unsure about '%S' (capital S)...

fixed

Ok, I implemented the changes that Johan proposed. Could we then get a review by a maintainer now and merge this stuff?

@Hackwar Thanks looking good.

One final remark. I'm not sure about the additional class properties that have been added without the underscore. I understand that in a future version the properties should not be underscored but adding them without using them makes little sense ?

Proposal

What about using the correct properties and removing the underscored ones and then dynamically assigning them again by reference when the object is instantiated ?

Example : $this->_vars =& $this->vars;

This should work, I think.

The benefit of this is that you don't need to add unused properties to the class definition. Keeping things nice and tidy. Removing the legacy support also becomes easier later.

Hi @johanjanssens, those properties are not added by me. That is stuff from 1.6 times, I think. In any case, the by-reference-trick will only work as long as no one assigns a new array to that property. I agree that we should get rid of that old crap, but I fear that it wont be possible before 4.0. Feel free to duke this out with the PLT.

@Hackwar You are right sorry, forgot to check the blame. This really smells bad. But if it was there, let's leave it there.

I updated the unittests to test properly for the new routing stages.

I had to change the order in which the methods are called in the router to solve a logical error that I made. So far the order was:

• Preprocess build rules
• Process build rules
• Call preprocess() of the component router
• Call build() of the component router
• Postprocess build rules

This would not allow the language system and the Itemid-lookup to work correctly and since the Itemid lookup should be pulled into the router, all plugins depending on the Itemid to be present in the buildRules step would fail subsequently, so I had to reorder this to the following:

• Preprocess build rules
• Call preprocess() of the component router
• Process build rules
• Call build() of the component router
• Postprocess build rules

It should be noted that the first order (preprocess() after buildRules) has been shipped with 3.3.6. So this would technically be a break in backwards compatibility if we change the order now. However, this requires that code in the preprocess() component router step has to rely on data from the buildRules step to even create an issue. This again means, that someone had to implement this so far undocumented and in the core unused feature. I would take the chance here and change this, especially since otherwise none of my routing changes can be merged before Joomla 4.0.

Imho we can take that B/C risk since I doubt anyone already used that new method. It would mean that they built an extension with a minimum requirement of J3.3.6 but they didn't really gain anything out of it.
Also there is a really slim chance if someone is using it, that the order actually matters.

Could we then merge this stuff?

@test
I did not get any issue on my multilingual test site.

@Hackwar The changes order for the build rules makes total sense and it more logical. Should the same change be made for the parse rules too to keep the implementation consistent ?

There is no preprocess step for the parsing for component routers, so there is nothing that we could change there.

True. All good then.