Sorry i wanted this PR for 4.3-dev - can anyone change this?

Means if a particullar string is not available in the main language it falls back to the fallback language? Or the whole file only?

Means if a particullar string is not available in the main language it falls back to the fallback language? Or the whole file only?

It tries to use an extensions language file as a possible fallback - if the extension provides a language file, it is loaded. this does not work on the string level

Would it not be better to enhance the current behaviour.

As it is today if the de-AT language file for the component does not exist then it falls back to en-GB

1. load de-AT
2. if not exist then load en-GB

I propose that instead the workflow is as follows

1. load de-AT
2. if not exist then load any language file beginning with de
3. if not exist then load en-GB

This way the "fallback" language does not have to be specified. It just follows a logical path

fr-CA --> fr-* --> en-GB

Would it not be better to enhance the current behaviour.

As it is today if the de-AT language file for the component does not exist then it falls back to en-GB

1. load de-AT
2. if not exist then load en-GB

I propose that instead the workflow is as follows

1. load de-AT
2. if not exist then load any language file beginning with de
3. if not exist then load en-GB

This way the "fallback" language does not have to be specified. It just follows a logical path

fr-CA --> fr-* --> en-GB

Thats a good point - we did indeed discuss an automatic solution, but i realized that this code was called very often and checking for an available language file may impact performance. On this level (not the string level) there is no fallback to en-GB (which wrongly would be en-EN) - Anyway i would like to change as little as possible - and there are not many countries needing this ;-)

Anyway i would like to change as little as possible - and there are not many countries needing this ;-)

exactly why I believe my proposal is the most efficient. (It's also the method used by other applications)

Another question. When a de-DE file and de-CH file exists are always both of them loaded?

And please update the test instructions so other can test it as well. As extension you can use for example DPCalendar as it comes automatically shipped with the de-DE language.

Another question. When a de-DE file and de-CH file exists are always both of them loaded?
Only the first available language is loaded, first language to test is the default language de-CH, then the fallback language.
(See the rather ugly early return at line 717 - existing code)

I have tested this item ✅ successfully on 64e89d9

Tested it with DPCalendar as it comes with de-DE shipped. After installing the de-CH language I set the fallback language to de-DE in the metadata file of the site language and DPCalendar opened successfully in German. Before the patch it was English.

I think it's the wrong direction. We should load a base language and override it with the localised Version.

I think it's the wrong direction. We should load a base language and override it with the localised Version.

You mean we should prepare (put in the array of files to be tried) automatically de-DE (derived from the first 2 characters in the language string) as well as the localised version (prior to the base version) without having the fallback tag?
We could do that - it changes the behaviour of actual installations however

I think it's the wrong direction. We should load a base language and override it with the localised Version.

Please add more information how you would do it. It is hard to understand what exactly you want to have differently. For the use case mentioned in this pr, this is a solid solution. Of course you can also do it like Brian mentioned but I like this approach more, so it is completely under control of the translation teams.

I have tested this item ✅ successfully on e9bd3a3

I have tested this successfully on localhost with JCE both on site and in the backend.

RTC

The thing is that normally you will install (we stay at the simple german or english version, based on my case) your local variant. This would be de-AT. the difference between de-AT and de-DE are only some marginal changes like ss and ß or Januar and Jänner. Now I have this shiny event component which has it's one translations for german and includes JANUARY as string which if we are optimistic is the only difference between de-DE and de-AT. So having all language string in de-AT a second time just for a single override is a type of overkill.

Other example why I think it's better to use the override approach is that your specific (or maybe own created) language file is out of date which means de-DE is already on 4.2.5 but de-AT is still on 4.1.5, this would mean for new strings it fallsback to english and not to the german german version.

Another part is this PR doesn't allow to fallback to de-DE from de-AT for joomla core.

Going the other way around could also be an option, the language pack can say which language tags it supports if no specific language is installed.
de-DE would support:

• de-AT
• de-CH
• de-LI
• ...

if you look at english then you have about 10 or 20 country/region specific packages.

last but not least, why only one fallback language, in case of de-LI I may prefer de-AT if installed and if not use de-DE.

I think i should call the new tag <extensionFallbackTag>

@HLeithner that is exactly why I proposed the different approach

Then @HLeithner and @brianteeman make a new pr so we can see the different approach. This one here fulfills the use case and is good for what it wants to improve.

Then @HLeithner and @brianteeman make a new pr so we can see the different approach. This one here fulfills the use case and is good for what it wants to improve.

that's your opinion, I don't think it's an improvement in the long run because so much more has to be fixed, just we have a pull request (thanks Martin) doesn't mean it have to be excepted (we talked about this on the maintainer sprint).

Merging this would prevent better/more complete solutions or at least creates more work on migration.

I don't think that this PR prevents better or more complete solutions. It is just a simple solution to a problem that special languages like de-CH have. One of the first things your hear in the swiss community is: Don't install de-CH as your favourite language because if you install extensions you will have them in english... (even if you have installed de-DE as an additional language and the extension provides de-DE)

de-AT has the same problem.
As mentioned above, fall forward is better then fall back in this scenario.
Load de-DE first and then if exists override with de-AT

So having all language string in de-AT a second time just for a single override is a type of overkill.

But this is how Joomla works now and out of scope of this pr.

Other example why I think it's better to use the override approach is that your specific (or maybe own created) language file is out of date which means de-DE is already on 4.2.5 but de-AT is still on 4.1.5, this would mean for new strings it fallsback to english and not to the german german version.
Another part is this PR doesn't allow to fallback to de-DE from de-AT for joomla core.

The problem is that a site doesn't have de-DE and de-AT installed at the same time, at least I see no reason to do so. So the different version of language pack issue doesn't count here and should be solved in a different pr when you want to fix this. This pr addresses a very common setup where people have only a variant like de-At installed but install an extension with only de-DE. All your other examples must be done in different pull requests and I doubt that this solution will block them in any way.

Going the other way around could also be an option, the language pack can say which language tags it supports if no specific language is installed.

Keep in mind an extension can ship it's language files in the installation bundle without any language manifest file. It is not an extra pack. Having this in the global language metadata definition, when a de-AT language also says, hey I'm de-CH as well, sounds for me like more trouble than it solves.

But this is how Joomla works now and out of scope of this pr.

it's simply the wrong approach from my point of view.

if you have a 10k lines files translation file for german and only 10 lines of austria dialect you think it's better to have 2x 10k lines. I don't think that this is right.

Also ignoring the core is not good, just saying out of scope makes no sense here, we have to check everything related.

if you have a 10k lines files translation file for german and only 10 lines of austria dialect you think it's better to have 2x 10k lines. I don't think that this is right.

Again, this is not the use case this pr tries to solve. It would impact much more changes in the whole eco system that extension developers ship only overwrites for dialects. This would also require changes in translation systems like Transifex or crowdin and it is unlikely that this will happen in the closer future. So there is no reason to shut down this pr which solves a real world problem we are facing as extension developer and site admin.

I'm not against the approach from @brianteeman. But somebody needs to do that pr as alternative to this one. If this is not the case, then this here is enough and should be merged.

The main (only) difference with my approach is that the change is only made in this file and it just works.

This pr also only makes a change in this file BUT it requires all the language packs to be updated. If a language pack is not updated which is the main reason for this pr then the user will see the strings in english.

In summary. This pr does not solve the problem

In the description is clearly mentioned that when this is merged, then the language packs are update afterwards.

Never mind, you clearly dont understand

Perhaps you don't express your self clearly.

This pr also only makes a change in this file BUT it requires all the language packs to be updated. If a language pack is not updated which is the main reason for this pr then the user will see the strings in english.

I will do another PR without using the fallbackTag and loading the main language first

This pr also only makes a change in this file BUT it requires all the language packs to be updated

It requires a change only in the language packs which want to support a fallback language. That's why I'm in favor to merge this early and update the language packs to have all ready for 4.3.

What I do not understand is, when you have so much resistant against this approach, why do YOU not do a pr with your own idea?

why do YOU not do a pr with your own idea?

If I had the skill I would

This pr also only makes a change in this file BUT it requires all the language packs to be updated. If a language pack is not updated which is the main reason for this pr then the user will see the strings in english.

That's correct and I also see your point with fr-CA --> fr-* --> en-GB but I think, that is not a desired behaviour.
First this function is opt-in (which I like) so if the swiss (de-CH) people decide, that they want de-DE as fall-back, they can set it like that. But I assume, that you don't want to have wild cards, because also some languages have the same base, they are not always compatible.

If you have de-CH as main language and the component author offers de-AT, I'm not sure if you want this as fall-back.

So you don't need to update all language packs, just the one who want to use this feature (de-DE will probably never have this tag)

That's why I'm in favor to merge this early and update the language packs to have all ready for 4.3.

Most of the language packs are not even upto date with the current release. Heck 30% are not even up to date with 4.2

It is only opt-in for the person making the language pack

Most of the language packs are not even upto date with the current release.

This doesn't mater for this pr, only the fallback tag needs to be added to the meta file. Nothing more.

If I had the skill I would

Then I apologize, I though you want to just block this pr.

Most of the language packs are not even upto date with the current release.

This doesn't mater for this pr, only the fallback tag needs to be added to the meta file. Nothing more.

Am I missing something. Isn't the meta file part of the language pack you install?

I think Harald idea to use Override is good.
Can be implemented similar to fallback, but with diferent order:

class Language
{
  ...

  protected $extendsLang = '';

  ...
  
  public function __construct(...) {
    ...
    $this->extendsLang = (string) $this->metadata['extends-language'];
  }

  public function load(.....) {
    ....
    
    $filenames = array();
    if ($this->extendsLang) {
      //... first add files for $this->extendsLang
    }
    //... then add files for requested language 
  }
}

Unless I something missed.

Having slept on it a bit longer I found a flaw with my proposal as opposed to this one.

Currently the fallback is fr-CA => en-GB
In my proposal the fallback is automatic and would be fr-CA => fr-* => en-GB

However I can see that with this PR it woul be possible to have fr-CA => de-CH => en-GB as it is manually created

I don't know if that situation that would be realistic but it woulld only be possisble with this pr.

I'm pretty sure that there are languages where the rule de-AT => de-* can't be applied. That's why, as we discussed the topic upfront, we decided to go with the metadata approach.

I approve this PR but I realized the fallback tag needs to be documented before I can merge it.

Where does it need to be documented? Is there some documentation for translation managers?

The language packs need to be created with the <fallbackTag/> tag. The language pack component should generate that tag, unless I am mistaken. It's not much that this PR needs to be documented, but language packs should now be able to contain the tag without needing a manual modification of the xml file. In other terms, these changes are going to be 'forgotten' if language files are not able to include <fallbackTag/>. Functionality will be there but not used. So there needs to be a 'follow up', maybe a PR for https://github.com/joomla-extensions/pkg_languagepack? Sorry, I am not expert on language package creation...

Only the official core languages can use that tag es they define the fallback. It is not must that they have it. The extension dev doesn't need to add that tag at all to their language.

The language pack component should generate that tag, unless I am mistaken.

There is no language pack component that would generate some tag, which as it is is the wrong solution in the first place to me too.

In other terms, these changes are going to be 'forgotten' if language files are not able to include .

The bigger issue is as soon as a better solutions without that tag, or a modified solution that also works for the core will be implemented that is going to be a big mess for b/c so it has to be fully documented on docs where the other language XML docs are placed and clearly mention what is intended, what are the ordering etc. Its going to be more of a problem too as core itself can not and will not use that tag in this setup for now. I still think we should think about a solution that can be applied to the core too.

Functionality will be there but not used. So there needs to be a 'follow up', maybe a PR for https://github.com/joomla-extensions/pkg_languagepack?

That component does not have anything todo with the generation and therefor a PR there does not help.

As long as this PR stays at it is it will always be a manuall task to be executed as the existing tooling would not output that tag, only none crowdin languages would have it easier as they could just workaround that and include the setting in a manuall step. Crowdin TTs need to manually alter the download comming from crowdin.

All of that should be documented on a docs page explaining what steps have to be done and what should be the consideration of using that tag and also that needs to be clear to the TT what is the result. The example used above with de-DE and de-AT is stright forward also as its comming from the same team. Not sure whether thats true for all other languages as well.

This is also going to be a B/C issue as it will massivly changing the behavior on a simple update. Right now the fallback in that case will be en-GB now in some places it will be de-AT or de-DE etc. To the regular user it is not understandable or even configuratable in the new setup other than hacking said component or revert this PR here. The other way arround would be a hacked TT package which is also bad.

I see the intention here is good but I would recomend to look deeper into this issue first and not just go with the first solution that comes into the mind and find a solution that can be applied on string basis similiar on how the fallback with en-GB is happening, and not redject comments with "it works as designed". Yes it works as designed but maybe we need a different design for the core setup.

Maybe its worth allowing that to be a setting, defauling to en-GB on user or even TT level but let that final decision of the user. That would allow the user to configure that setting and also allow us to perseve B/C for upgrades. Ontop on that I would say this should be string based too but said setting would allow full controll to the user and siteowner to do that fallback what makes sense to thier site. Maybe some kind of "language pack suggested" information can be provided but not forced to all people using that language pack i would say.

I agree that for the core, we should place the decision in the hands of the user.
As a third-party developer, I see it as an advantage if a language is missing in my extension and that the user can get a 'close alternative' that I can provide.
I can see valid points here but, in the end, let's make sure the user is the one benefiting from this.

I agree that for the core, we should place the decision in the hands of the user.

This here is a change to the core which has b/c issues and issues resulting into major changes to the users they dont have a way to control.

As a third-party developer, I see it as an advantage if a language is missing in my extension and that the user can get a 'close alternative' that I can provide.

Again I fully agree about the problem statment, is something I was thinking about looking into myself too a few times but never came to look into the code. But this solution here only makes matters worse by not allowing the users to decide whats happening. We have to think about the core to not just a workaround for extensions.

I can see valid points here but, in the end, let's make sure the user is the one benefiting from this.

I do too but I dont think that we should do this by forcing a major changes over them they have not control over to disable and changing an important part of joomla just half way in a minor update.

On the extension side of things there is no need to relay on core changes when you want to ship the de-DE strings for de-AT go and copy the de-DE file to de-AT, problem solved and your users are happy. But when we do a change in the core including chnages to translation packs we should think about all consequences and find a solution that has been checked and verified to be sutibel for the core. Translations are a big part where we in Joomla are good its not something that we should take easy and just apply major changes without any options aviable to the users.

@zero-24 feel free to make an alternative pr.

On the extension side of things there is no need to relay on core changes when you want to ship the de-DE strings for de-AT go and copy the de-DE file to de-AT, problem solved and your users are happy

If you would maintain an extension which thousands of strings, then you would see that this is not an easy task. Especially when you use translation plattforms like crowdin or transifex.

I really see no point in making this another option which nobody understands.

If you would maintain an extension which thousands of strings, then you would see that this is not an easy task. Especially when you use translation plattforms like crowdin or transifex.

A one to one copy of a translation file? You are using a build script right? What holds you back from adding one cp line? The tools mentiond should make it even easier and you know as good as me that the count does not matter much here as all the translations does not need to be managed by you anymore. You could add to your translation sync script the required cp statments and after that its also fully automated, cp de-DE to de-AT upload the strings, download the strings done. I do agree there are some management required when you actually want to manage different languages but this PR does not solve that anyway its a 1:1 copy/fallback for the complete file to de-DE too.

feel free to make an alternative pr.

Yes as mentiond in my comment its on my list already but i have not yet had the time to do a PR for that.

In general I like to have a fallback language in Joomla, especially for extensions. Like every Swiss Joomla beginner a had hard times as I started Joomla to find out what to do in order for extensions to show up in German. Of course I had installed de-CH back then like every Swiss newby will probably do. So thank you very much @MacJoom for this solution.

I see this discussion heading towards a question of design principle...

• Is Joomla designed to be customized/controlled as much as possible by the user. Then I would suggest to add some kind of setting in the Joomla global configuration where the user can decide a fallback language by its own.
  (This is good for advanced users who want the freedom of customize everything to their taste.)

• Or is Joomla designed to be very easy in usage. Then I like this approach since every translation team decides for its userbase.
  (This principle is good for users who want the system to run with as few clicks as possible.)

Do we have some general design priciples from Joomla which can be used to make the decision? Something like "We want to have a CMS which is highly customizable." or "We want to have a CMS which is as easy to set up and use as possible."?

Or is Joomla designed to be very easy in usage. Then I like this approach since every translation team decides for its userbase.
(This principle is good for users who want the system to run with as few clicks as possible.)

That solution is not only that, its breaking B/C on a update just because we can, that current code does not support core translations but only extensions and is adding a setting which is going to cause more complex coding to keep the than old behavior once a better solution has been build. ;)

When an extension developer like to implement the changes proposed here go for it just copy the de-DE file to de-AT (and similiar languages). But when something is implemented into the core we should think about all places where this has an impact.

I does not have to be a setting by the user, its just that I think we should not force it on updates and I personally think it should be configurable. But if anything we should not break b/c as well as not merge something thats only half way done and not transparent to the user.

@zero-24 what breaks here bc?

two ways. Sites that have setup an extension thats expected to show en-GB strings for de-AT or where de-AT overrides for en-GB have been created to mitigate the issue. As well as this is changin stuff thats inconsistend for core and some extensions too without any indication to the user that and why its changed.

This pull request has been automatically rebased to 5.1-dev.

So how are we going on here?
Its out of discussion that a fallback possibility is needed, but we havent found yet the best way to do it right?

This pull request has been automatically rebased to 5.2-dev.

Has this been abandoned? If so then it should be closed

Hopefully not. It would be a very important feature!

Well I do still stand by my points mentiond above. If anything it should not be automagic with no indications of what is happening here in the background and there should be an option to disable this feature. I would say such a setting could be part of the multilanguage settings where you can configure which language should be used as a fallback for example de-DE for de-CH frontend language.

As for extensions when you dont have a dedicated de-CH translation it should not be complicated to copy the files from de-DE to a de-CH folder -> Done thats a one liner in the build script and with the new filename that dont have to include the language tag anymore it could not be easier to solve that issue. As that is also a solution which works for all versions of Joomla too ;)

thats a one liner in the build script and with the new filename that dont have to include the language tag anymore it could not be easier to solve that issue.

Happy to test a PR with this approach. Till then I guess this here is the one approach we have a solution for...

thats a one liner in the build script and with the new filename that dont have to include the language tag anymore it could not be easier to solve that issue.

Happy to test a PR with this approach. Till then I guess this here is the one approach we have a solution for...

Do you have an extension build script for me to send a PR against? When a de-DE folder exists there is no problem to copy this folder to de-CH within that script -> Done.

Do you have an extension build script for me to send a PR against? When a de-DE folder exists there is no problem to copy this folder to de-CH within that script -> Done.

Thats defenitely not a nice solution. Build script only make sense when developing big extensions. For Modules and Plugins I defenitely not doing a build script. Thats overkill.

Well it do not have to be a "big" build script. I'm using a very simple one that also generates the hashes for the files and excludes the files from the repo which should not be packaged: https://github.com/zero-24/plg_task_extensionupdates/blob/master/build/build.sh

When that is done all manuall there is also no problem to copy de-DE to de-LU etc. Usually translations are also not changing massivly for such simple modules or plugins so its even just a onetime job.

Spoiler even for the core we just copy de-DE to de-LU, de-AT etc https://github.com/joomlagerman/joomla/blob/5.2-dev/build/build.php#L150 ff with very little changes for each language: https://github.com/joomlagerman/joomla/blob/5.2-dev/build/build.php#L361 ff

Build script only make sense when developing big extensions. For Modules and Plugins I defenitely not doing a build script. Thats overkill.

I have build scripts for all extensions, starting by for a simple 2 line module and ending by a template, in between is everything von plugin and library. It's automatically generated, I use Jorobo for it that means I only have a robo.ini, minimal RoboFile.php and a composer.json that's it.

The build script is the smallest part of all the code that lays around in every extension:

So instead of providing a robust solution, you expect every extension developer to make a build script and copy files back and forth? Don't think so that this is sustainable.

I suggest to rebase this one to 6.0 and then merge it as this is clearly a big improvement for extension developers which do distribute extensions for the public and users which do use Joomla outside of main language areas.

I do disagree that we should just merge this into 6.0-dev as explained above. I have outlined the proposed longterm solution upwards. I have said the copy thing is the short term solution when you have that big problem now and want to have a short term solution. But we can not put something in place that no user can control and by default changes the behavior.

I suggest to rebase this one to 6.0 and then merge it as this is clearly a big improvement for extension developers which do distribute extensions for the public and users which do use Joomla outside of main language areas.

Merging this as is will surely break existing/expected behaviour. At a minimum I would expect a site owner to have the ability to set the fallback language to use - if they want to use something other than the default english when their selected language is not available for that extension. It might make sense for a TT to decide on de-DE for the de-AT language pack but in other cases less so.

With the option to set my own fallback language I could for example install the klingon language pack and when dp-calendar doesnt provide a klingon translation chose to use the romulan translation instead (and failing that to use en-GB). Another site owner might chose to use the ferengi translation instead as a fallback

At a minimum I would expect a site owner to have the ability to set the fallback language to use - if they want to use something other than the default english when their selected language is not available for that extension.

However, this would undermine the initial problem that this PR wants to solve.

The thing is that an inexperienced user who has e.g the 'de-AT' language pack activated is surprised and confused why the extension he just installed presents him an english site even though the extention developer explicitely stated that the german language is available.
If this user knew about Joomla's language fallback system, he would also know how to solve this problem. But this PR is not for such users. It is ment for Joomla beginners or inexperienced users who are not aware of this fallback mechanism and will not be able to find such an option where they can set their preferred fallback language. Therefore this must be defined by the developer of the extension, otherwise the original problem is not solved.

@zero-24 suggestion of copying language files would work, but this doesnt look like a proper solution to me.

@zero-24 suggestion of copying language files would work, but this doesnt look like a proper solution to me.

Yes thats the reason i proposed it as shorterm solution, it solves the problem extension devs have right now and this is even compatible down to 2.5 etc ;)

It is ment for Joomla beginners or inexperienced users who are not aware of this fallback mechanism and will not be able to find such an option where they can set their preferred fallback language

The solution here forces it for every user without any controll by the site owner and also massifly changes the behavior on existing sites. Its not something just for the inexperienced users as this PR here is.

If this user knew about Joomla's language fallback system, he would also know how to solve this problem

How would they do that. There is NO system to solve this

Therefore this must be defined by the developer of the extension, otherwise the original problem is not solved.

This proposal has nothing to do with extension developers

This pull request has been automatically rebased to 5.3-dev.

The solution here forces it for every user without any controll by the site owner and also massifly changes the behavior on existing sites. Its not something just for the inexperienced users as this PR here is.

Existing sites do not change their behavior. Only if an update to the an extension is installed which introduces the extensionFallbackTag - the language on this extension changes to the fallback language. Its all up to the extension developer.

The solution here forces it for every user without any controll by the site owner and also massifly changes the behavior on existing sites. Its not something just for the inexperienced users as this PR here is.

Existing sites do not change their behavior. Only if an update to the an extension is installed which introduces the extensionFallbackTag - the language on this extension changes to the fallback language. Its all up to the extension developer.

This is not what i see in the code of this PR. The code here asks for an change within the upstream language package. So de-AT has to set de-DE as fallback and with that change the behavior will be changed for all users installing that new language pack and has an effect on all extensions.

Take a look at your own test instructions above its not a per-extension setting. ;⁠-⁠)

Sorry my bad - you are right. Its been a long time since I made the PR. I will change this. I like to do it the way I described it in my last comment

No problem, i do still think that a simple copy of the language folder is easier for extension devs. But when you do a new PR we can close here right? So we get a fresh PR with a mention back to this PR ok?