IMHO i think they should be removed in the next minor release if not used.

And lower joomla versions should not update the language package to a release upper than it's joomla release.

I agree - its crazy that a new language has to translate so many unused
strings

Questions
1. We would then need a way to mark a language tag as unused since version
x.y.z
2. We also need to do something now to mark those strings that are already
unused and not just for future strings

On 15 April 2016 at 13:42, Robert Deutz notifications@github.com wrote:

Situation today

Unused language tags aren't deleted and stay for an long time within the
language files. This makes the management more complicated and while adding
a new language we can't easy see what has to be translated and what is
already not used.
Proposal

Unused language tags stay for 2 minor releases in the language files and
then they will be removed.

Example

• Langtag not used in 3.4.x
• Stay for 3.4.x, 3.5.x, 3.6.x
• Deleted in 3.7.0 (or 4.0 whatever comes first)

Reasons

• Better management of language tags
• People will not only update the language files and not the core CMS over a couple of minor Version. And if, it might be a sign that they need to update.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#9932

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

IMHO it's an over complicated process that could be simply resolved by not allowing old joomla version to update their language packs to a release upper than it's joomla release.

Can anyone tell me what is the problem with this approach?

If they want to update the language packs ... update joomla first!

Nothing - it is brain dead that we allow someone on Joomla 3.3 to use a
language pack for 3.5 and expect it to work. There is no logical
justifcation for this. But I hit my head on too many walls

On 15 April 2016 at 13:48, andrepereiradasilva notifications@github.com
wrote:

IMHO it's an over complicated process that could be simply resolved by not
allowing old joomla version to update their language packs to a release
upper than it's joomla release.

Can anyone tell me what is the problem with this approach?

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

i agree joomla should allow install lower language versions on higher Joomla versions of course. But the opposite does not makes sense to me.

I, make a change to my comment, as i can also agree in deprecating and remove it only in the next minor release so extension can have time to check if they are using some of those deprecated language vars.

Questions
1. We would then need a way to mark a language tag as unused since version
x.y.z
2. We also need to do something now to mark those strings that are already
unused and not just for future strings

re 1: we could move them down to the bottom of the lang files after a ; ##### UNUSED SINCE 3.4.3 #####

re 2. I have written a script to check all files and language tags if they are used or not, it isn't perfect because of the way some language tags are created but gives you a good idea

re 2 - there is a script already from @adaneinspain sounds like you have
the same problem with combination strings as him

On 15 April 2016 at 14:18, Robert Deutz notifications@github.com wrote:

Questions
1. We would then need a way to mark a language tag as unused since version
x.y.z
2. We also need to do something now to mark those strings that are already
unused and not just for future strings

re 1: we could move them down to the bottom of the lang files after a ;

UNUSED SINCE 3.4.3

re 2. I have written a script to check all files and language tags if they
are used or not, it isn't perfect because of the way some language tags are
created but gives you a good idea

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

TBH... both scripts (sorry @rdeutz ) are far from perfect in both ways... finding strings not deprecated and not finding some that are.
It is a real nightmare, believe me.

I've already said in the past that if we patch up https://github.com/joomla-projects/language-update-xml-generator correctly that newer language packages would stop showing up on older releases, but thinking about it some I honestly don't think the update system can handle it.

You have to account for the fact that not every 3.x version has the same features in the update system (IIRC 3.0.1 or 3.0.2 added the min_dev_level and max_dev_level attributes on the <targetplatform> tag and 3.5.0 added support for making the version attribute regex work with the full CMS version (3.4 and earlier will only check up to the minor version)). Also the *_dev_level attributes won't work effectively with a regex (if I have a 3.[012] version regex and max_dev_level attribute set to 4, it effectively limits the versions something can install on to 3.[012].[0-4].

Then you have to be able to show the correct version in every release. And that's where the trickery or ugliness starts. Essentially https://update.joomla.org/language/translationlist_3.xml would have to list every language package release for the entire 3.x series (not that big of an issue to do, it's just code generating it). But doing that IIRC would affect what's displayed in the language install screens for the install app and extension manager, so without a core patch to filter results those lists are going to go from ~60 items to a couple thousand in a heartbeat.

So long and short, at least from the update system's perspective, I don't know what we can do to filter/limit results and I doubt the TT's are going to impose a script.php file requirement on the packages to check versions on preflight and block installs that way.

@MATsxm that's fine, just spend some hours and to have a good environment to test is a challenge in itself

Worked HOURS on this question and it is a real nightmare... it's easier when fr-FR are still supporting language packs for GeSHi... you easely know then what to delete

Also the question about merging some sys.ini with .ini that are the same.

@mbabker showing them is only one problem, but I can also download the package and install it, so without some kind of dependency management that will not work.

That's why I tried to get the "low hanging fruit" and solve the problem on the organisation level :-)

but I can also download the package and install it, so without some kind of dependency management that will not work.

Can't the package manifest have a limit in version too? Even if not, is a Joomla problem if a user manually does that?

I can also download the package and install it

If you're going out of your way to download and install something that the core UI is filtering out as non-supported on your platform you deserve all the untranslated strings that come with it ????

@infograf768 I would like to hear your opinion on this proposal, thanks.

Imho it boils down to the updater as said already. As long as our language manager proposes to install the language pack for 3.5 in a 3.3 install, I wouldn't delete the strings. You would end up with incomplete translations.
Since Michael thinks our updater can't handle it properly, I would look to change it fo J4.

As for deprecating, I wouldn't move the strings around. You could just add a comment behind the string (on the same line) and translation tools like Transifex and Crowdin would pick that up and show it to the translator.

Why should we care if someone installs a language file for 3.5 into an
unsupported version of joomla

On 21 April 2016 at 21:23, Thomas Hunziker notifications@github.com wrote:

Imho it boils down to the updater as said already. As long as our language
manager proposes to install the language pack for 3.5 in a 3.3 install, I
wouldn't delete the strings. You would end up with incomplete translations.
Since Michael thinks our updater can't handle it properly, I would look to
change it fo J4.

As for deprecating, I wouldn't move the strings around. You could just add
a comment behind the string (on the same line) and translation tools like
Transifex and Crowdin would pick that up and show it to the translator.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

Because we don't intentionally want to break existing sites.

We wouldnt be - the user would be breaking their own site by chosing to
upgrade a language pack without upgrading joomla

On 21 April 2016 at 21:36, Thomas Hunziker notifications@github.com wrote:

Because we don't intentionally want to break existing sites.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

@Bakual it seems we can't fix it technically in a overseeable future so I looked for a solution on administration level

How is a site being broken? Because a string isn't localized anymore? They'll still have the English string as a fallback. Considering we have language packages that AREN'T offering a 3.5.1 version it's safe to assume then anyone using old language packages on a newer version of Joomla have broken sites by that standard.

Using outdated language packs on a newer Joomla installation is another topic. Not much we can do there.
Using newer language packs on an outdated Joomla installation is the topic here.
Yes it would fall back to english (is that the case for whole 3.x? Don't remember), that's true. However I don't see that as a desirable effect.
Yes, they should update their installation, but we all know there are outdated sites out there.

Another point is that those strings also could be used by 3rd party extensions which would leave them with untranslated keys even when the installation is up to date. There wouldn't even be an english fallback.

Imho, the risk isn't worth the reward of having less to translate. Especially since we could already mark the unused keys as deprecated and translators would be able to skip them if they wish to do so. So basically we could have the reward already without any risk involved. Why remove them then within 3.x?

Using outdated language packs on a newer Joomla installation is another topic. Not much we can do there.

Except it's relevant and pretty similar behavior except switching where the words "outdated" and "newer" are in the sentence. They still fall back to English if the key doesn't exist or the raw key if strings were ever removed just like a newer version would.

The difference is that with an outdated language pack you don't suddenly loose a translation when you update. While in the other case they will loose translations when they updtae the language pack (which is completely unexpected).

Again why should we care if someone updates the language pack but does not
update joomla.

On 22 April 2016 at 09:13, Thomas Hunziker notifications@github.com wrote:

The difference is that with an outdated language pack you don't suddenly
loose a translation when you update. While in the other case they will
loose translations when they updtae the language pack (which is completely
unexpected).

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

Agree with moving obsolete strings after a line like this:
[Obsolete strings]
Adding between the brackets since Joomla version # would create more issues than solved as one is never sure of the milestone.

Just to be clear @infograf768 you are suggesting that all unused strings
are moved to the end of their respective language file after a tag to show
they are obsolete and without any information about the version they were
removed in (I agree that could be hard historically). Do they still need to
be alpha ordered

On 22 April 2016 at 09:59, infograf768 notifications@github.com wrote:

Agree with moving obsolete strings after a line like this:
[Obsolete strings]
Adding between the brackets since Joomla version # would create more
issues than solved as one is never sure of the milestone.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

if we do not add a j version, alpha order them there looks logical.

Using brackets[...] instead of a comment ; ... will make them stand (and in com_localise, they will show in a specific slider)

hum brackets [...] is a ini file section. sure will not have problem with that?
See https://en.wikipedia.org/wiki/INI_file#Sections

Yep, totally sure.

ok then

If we just move the unused strings to the end with a comment above the block will any of the translation tools see that these dont need to be translated. If not then there is no point

Just using a bracket title and moving them to the bottom will not solve anything for translators using Transifex/Crowdin/OtherTool. It only helps those who translate directly from the file (or use com_localise, our homebrewed translation component)

For those tools you need to add a comment on the same line after the string, then it will be shown as context.

You can hide them on Crowdin at least... not a big deal if you know the ones to.

I suspected that might be the case

to clarify and on Crowdin (I didn't check the other platforms).
-You can make them NOT visible for translators (only managers can see them).
-Indicate a context and thus that a specific string must NOT be translated or as "optional" etc.
-Integrate them in a glossary to clarify that they should not be translated (we can specify for some specific languages only and I tested it without finding the limit of the number of characters then you can add long strings).

You can hide them on Crowdin at least... not a big deal if you know the ones to.

Manually, yes.

Indicate a context and thus that a specific string must NOT be translated or as "optional" etc.

If we mark the strings in the source file like this:
EXAMPLE_DEPRECATED_KEY="Some deprecated stuff" ; Deprecated since 3.5.0
Then that comment will show as context in Crowdin (and Transifex afaik)

You can hide them on Crowdin at least... not a big deal if you know the ones to.
Manually, yes.

This is really not a big deal... even I did it on other projects

Unused and obsolete strings has nothing to do in the core language files,
in the same way as unused and obsolete code has nothing to do in the core
package.
With +250 language files and thousands of strings, it should at all times
only be valid and in use strings present in the files, for the current
release.

Volunteer translators and language maintainers should not worry about how
people use joomla the wrong way ~ outdated versions.
We expect and recommend users to always use latest release of joomla, which
include to use latest release of the language packages.
So a promt delete of an unused string when found should be the standard,
IMO :)

On Fri, Apr 22, 2016 at 9:29 PM, Marc-Antoine Thevenet <
notifications@github.com> wrote:

You can hide them on Crowdin at least... not a big deal if you know the
ones to.
Manually, yes.

This is really not a big deal... even I did it on other projects [image:
]

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#9932 (comment)

+100

On 22 April 2016 at 22:26, ot2sen notifications@github.com wrote:

Unused and obsolete strings has nothing to do in the core language files,
in the same way as unused and obsolete code has nothing to do in the core
package.
With +250 language files and thousands of strings, it should at all times
only be valid and in use strings present in the files, for the current
release.

Volunteer translators and language maintainers should not worry about how
people use joomla the wrong way ~ outdated versions.
We expect and recommend users to always use latest release of joomla, which
include to use latest release of the language packages.
So a promt delete of an unused string when found should be the standard,
IMO :)

On Fri, Apr 22, 2016 at 9:29 PM, Marc-Antoine Thevenet <
notifications@github.com> wrote:

You can hide them on Crowdin at least... not a big deal if you know the
ones to.
Manually, yes.

This is really not a big deal... even I did it on other projects [image:
]

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
<#9932 (comment)

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

Unused and obsolete strings has nothing to do in the core language files,
in the same way as unused and obsolete code has nothing to do in the core
package.

If we threat it the same as the code, that would mean proper deprecate and removal with 4.0, and is what we do as of today and follows SemVer.

Why is there such a strong insistence that language packages are 100% B/C forever? I can install the 2.5 packs on 3.x, and the 3.x packs on 2.5. Why doesn't Joomla support that too and go ahead and pull back all the 2.5 strings that don't exist in 3.x? IMO the "it doesn't show up in the install list so users obviously won't do it" argument is invalid; some comments up we talk about users being able to do just that even if it's figured out how to filter that screen.

Somewhere along the line you have to draw a line. You have translators continuing to translate strings which are not part of the core package for the sake of someone still running Joomla 3.1.1 being able to install the Joomla 3.5.1 language package and not have an untranslated string. Why?

I've been told the GeSHi strings are still included in some language packages. That plugin isn't even part of 3.x yet is found in 3.x packages. Why?

Personally I look at language packs the same as I do at my extensions.
I have to decide what my extensions minimum requirement is and enforce it. If I want my extension to be installable in 3.0 up to 3.5 then I need to make sure it's compatible with all versions. Or I can take the easy route and say my extension will only run at the latest version and enforce it in the Joomla Updater (and install script).
An extension which only runs with the latest Joomla version but offers an update and is installable on all previous versions as well just doesn't look like it's properly done to me.
Same with language packs. If the update shows and can be installed, I just don't expect it to "break" things. And it also never did so far. So if suddenly translations will be missing, that would look like a bug to me.

Let's start marking strings as deprecated so we see what we are talking about. Amount and type would be important factors to make a decision.
Also we need to be 100% sure the string isn't used anymore. Which isn't that trivial to check as it may sound (for some strings, it will not show up in a simple search).

I've been told the GeSHi strings are still included in some language packages. That plugin isn't even part of 3.x yet is found in 3.x packages. Why?

I think that's the French language which has it still. I'm not sure if there are other languages still translating that one. At least not those using Crowdin

Or I can take the easy route and say my extension will only run at the latest version and enforce it in the Joomla Updater (and install script).

That's not the easy way. That's the way it should always be done (replace "latest" with "minimum"). So in theory you need minimum Joomla 3.5.1 to use the 3.5.1 language pack. That CAN be enforced easily with a script.php file, but it mandates then that TT's update a second PHP class and I've heard of general reluctance to mandate TT's work with PHP.

What can't be done easily is dealing with the updater. Sure, we can patch it for newer versions, but what's there now is probably the best you're going to get for the already released 35+ 3.x releases. And I don't agree that because the updater most likely cannot effectively block seeing newer releases is a reason why 3.x language packs need to be 100% B/C to 3.0.0 (FWIW before 3.0.1, which includes all 2.5 releases since the change wasn't backported, you couldn't even filter a release to not be displayed on specific patch versions with the updater and I will still say that this technology limit shouldn't be a primary reason why you don't do something when the updater is involved). It's mainly the collection adapter that's the issue; from what I know the single extension adapter would handle our UI requirements just fine.

TL;DR... The update system's limitations and not using common subsystems such as script.php files shouldn't be the primary reason you say "lang packs must be 100% B/C forever".

That's not the easy way. That's the way it should always be done (replace "latest" with "minimum"). So in theory you need minimum Joomla 3.5.1 to use the 3.5.1 language pack. That CAN be enforced easily with a script.php file, but it mandates then that TT's update a second PHP class and I've heard of general reluctance to mandate TT's work with PHP.

I agree with that. In my own extension I use a simple code block in the preflight method which checks the version attribute in the XML and if the current Joomla version isn't compatible with that attribute it returns false, terminating the installation.
Eg https://github.com/Bakual/SermonSpeaker/blob/master/com_sermonspeaker/script.php#L47-L58
We could so something similar for the language packs and translators wouldn't have to deal with PHP at all since you don't need to change anything there. Just adjust the attribute in the XML (which can be copied from the en-GB file.)
For those languages using Crowdin as the translation tool, that would be simple to implement.
For those using com_localise it should be doable as well I think.
For those using other tools or an editor, manual work would be involved.

@Bakual
What kind of attribute would it be?
And where? In the pkg_xx-XX.xml or both install.xml?
Would it have any effect for the list of languages proposed to install/update in J UI?

I'm not sure if the package installer supports a script file. I would use the one from the languagepack itself, eg in https://github.com/joomla/joomla-cms/blob/staging/administrator/language/en-GB/install.xml#L2 the version attribute.
If we would only remove the keys with a minor release, then it wouldn't change anything for translators as they are supposed to change that version attribute anyway to the current minor release (as far as I know).

I'm not sure if the package installer supports a script file.

The package installer has supported script files since Joomla 2.5.0.

Great stuff. Then we could put the scriptfile into the package and use the package version attribute (https://github.com/joomla/joomla-cms/blob/staging/administrator/manifests/packages/pkg_en-GB.xml#L2).

We can easily get excited about what is possible to do with joomla. That
far I can agree :)
There are things that shouldn´t be done, just because we can though.

I hear you Thomas, and will have to clarify that my comparison of unused
code vs unused language strings wasn´t optimal.
The language strings can´t be seen as deprecated as code functionality can.
When it comes to language strings they are not deprecated from x point of
time, they are directly unneeded the moment their language key is no longer
in the code.

Should we decide to allow unused keys/strings to stay, we are encouraging
the opposite of our standard by saying it is ok to use an outdated joomla -
it is not.
Users who did upgrade a language package and now maybe see either an
English string or the language key in their interface, well they only get a
very direct reminder that they should update the code part asap too.

Let me repeat, for maintenance and for translating (and for reporting back
potential localisation issues which would need adjustments to the
internationalisation part of the code/visuals) then having x % unused
strings surely isn´t optimal. With 6-7000 strings and some undefined amount
of unused, could be 5-10%, we are not helping new language teams, nor
current language teams, in doing their best of translating and localising
the software.

Please use the standard saying: Always use latest release.

On Sun, Apr 24, 2016 at 8:44 PM, Thomas Hunziker notifications@github.com
wrote:

Great stuff. Then we could put the scriptfile into the package and use the
package version attribute (
https://github.com/joomla/joomla-cms/blob/staging/administrator/manifests/packages/pkg_en-GB.xml#L2
).

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

When it comes to language strings they are not deprecated from x point of
time, they are directly unneeded the moment their language key is no longer
in the code.

They may be used by 3rd party extensions even if they are not used by our own code. So far it's actually similar to code.
Eg a plugin or module related to com_content may use language files from com_content to reduce own translation work.
Or of course if we talk about the common files (library one or the en-GB.ini) the strings are used by many extensions.
Just removing strings without any warning at all isn't optimal and completely unexpected in that case. And it doesn't even need any outdated CMS/extension here.

It is understandable to have a wish to reduce own translation workload from
an extension developer point of view.
Reuse of core strings in a related extension could be tempting, but it
isn´t optimal for more reasons, a couple of these could be:

1. For the developer they suddenly could have a string in their module of plugin that was changed in a core language pack update and now doesn´t give the correct meaning in that related extension.
2. For site integrators and site maintainers they might wish to override a string in your extension, and get negatively surprised that they unaware did override a string used elsewhere in the core component and that this core text now doesn´t present what it should do.

Standard would again be that extension developers always use unique strings
for their unique extension.
(only exception would be the few JNO, JYES, etc. the so called global
strings, which are of a very limited amount. These we do recommend
developers to use)

On Sun, Apr 24, 2016 at 11:15 PM, Thomas Hunziker notifications@github.com
wrote:

When it comes to language strings they are not deprecated from x point of
time, they are directly unneeded the moment their language key is no longer
in the code.

They may be used by 3rd party extensions even if they are not used by our
own code. So far it's actually similar to code.
Eg a plugin or module related to com_content may use language files from
com_content to reduce own translation work.
Or of course if we talk about the common files (library one or the
en-GB.ini) the strings are used by many extensions.
Just removing strings without any warning at all isn't optimal and
completely unexpected in that case. And it doesn't even need any outdated
CMS/extension here.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Reuse of core strings in a related extension could be tempting, but it
isn´t optimal for more reasons, a couple of these could be:

Not arguing with you as I agree. However I'm quite sure it is done in the wild and we should not break those extensions.

By that argument we should never change an existing string as that would
break the extension of a lazy developer who has re-used them. I know for a
fact that I have changed strings which if used elsewhere would have
"broken" another extension that used them

On 25 April 2016 at 11:17, Thomas Hunziker notifications@github.com wrote:

Reuse of core strings in a related extension could be tempting, but it
isn´t optimal for more reasons, a couple of these could be:

Not arguing with you as I agree. However I'm quite sure it is done in the
wild and we should not break those extensions.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#9932 (comment)

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

I'm not saying not to change. But we have to be aware of it.
Personally I think changing is fine, it doesn't leave the extension with untranslated ugly strings. Removing is also fine if

• we mark them somehow as deprecated prior to removal so people at least have a chance of knowing (deprecate notice) and remove them only with minor versions, not patches.
• we take care that the language pack isn't installed on older Joomla versions (scriptfile).

I'm not a fan of just directly removing strings.

I'm not a fan of just directly removing strings.

Nobody's trying to imply just remove them and be done with it. Yes, there are a lot of details to figure out along the way, yes there are technical issues to address, no the status quo shouldn't be retained by the time it's all said and done.

I think everyone who is likely to comment has expressed their views by now and this can be closed and handed back to PLT for a decision