?
avatar Radek-Suski
Radek-Suski
18 Apr 2017

While testing Joomla! 3.7 we realised that many of SobiPro language files are not being loaded. I debugged it and according to PHP some of our files are not valid because, for example, the value contains brackets.

The solution would be to set the scanner to "raw" while using the parse_ini_file function.
I am pretty sure we are not the only ones using brackets in language string.

Speaks anything against using the raw scanner?

libraries/joomla/language/language.php:833
$strings = @parse_ini_file($filename,false,INI_SCANNER_RAW);

avatar Radek-Suski Radek-Suski - open - 18 Apr 2017
avatar joomla-cms-bot joomla-cms-bot - labeled - 18 Apr 2017
avatar brianteeman
brianteeman - comment - 18 Apr 2017

did this work before 3.7?

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

Yes

avatar brianteeman
brianteeman - comment - 18 Apr 2017

Related to this PR #13407 perhaps

avatar zero-24
zero-24 - comment - 18 Apr 2017

hmm the only change done there was:

old

$contents = file_get_contents($filename);
$strings = @parse_ini_string($contents);

new

$strings = @parse_ini_string($filename);

Could this be the issue? can you check @Radek-Suski ?

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

I don't think so. I had this already with the first beta but I thought it something I screwed in my code. I was searching in 3.6 and there is no parse_ini_file at all. Unless I am blind

avatar franz-wohlkoenig franz-wohlkoenig - change - 18 Apr 2017
Build RC2 3.7-rc2
avatar franz-wohlkoenig franz-wohlkoenig - change - 18 Apr 2017
Category com_languages
avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

No, sorry. I was wrong. So I have no idea why it worked before

avatar zero-24
zero-24 - comment - 18 Apr 2017

did you try the change from above? So you have the string vs the file name as parameter?

avatar brianteeman
brianteeman - comment - 18 Apr 2017

it cant be because of the brackets as many core strings have brackets

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

Ahhh. Let me check

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

No, parse_ini_string doesn't work for me either.

root@sigsiu-pi:~# php -r "parse_ini_file('/www/Neo/language/de-DE/de-DE.com_sobipro.ini');" PHP Warning: syntax error, unexpected '(' in /www/Neo/language/de-DE/de-DE.com_sobipro.ini on line 26 in Command line code on line 1 PHP Stack trace: PHP 1. {main}() Command line code:0 PHP 2. parse_ini_file() Command line code:1

This is really weird.

avatar zero-24
zero-24 - comment - 18 Apr 2017

did the exact old code work?

$contents = file_get_contents($filename);
$strings = @parse_ini_string($contents);
avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

Ok. I think I see what's the issue is. When I restore the code back:

`
$contents = file_get_contents($filename);
$contents = str_replace('QQ', '"""', $contents);
$strings = @parse_ini_string($contents);

`

It works fine. I have QQ in this file so I guess it is rather this problem and not the brackets

avatar brianteeman
brianteeman - comment - 18 Apr 2017

we still have QQ in the core

avatar zero-24
zero-24 - comment - 18 Apr 2017

@Radek-Suski did it also work if you remove the seccond line and just use the code i proposed above?

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

No. It doesn't work then. Let me check Brian. Maybe it is something wrong with our files then

avatar brianteeman
brianteeman - comment - 18 Apr 2017

I would take a look but sobi wont install for me at all

Please wait, checking requirements ... Wait please ... 1 processes are still running

avatar mbabker
mbabker - comment - 18 Apr 2017

parse_ini_file() respects constants that have been loaded when a file is parsed. So the manual parsing/replacing that was used in 3.6 is unnecessary.

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

Ok, I think we found it. The problem with QQ is that TransiFex apparently is adding some strange construct to it:

SP.IMAGE_UPLOADED_CROP="Die Datei \"QQ"%s\"QQ" wurde hochgeladen. Klicke auf die Bildvorschau um den sichtbaren Bereich anzupassen."

It worked with the previous method (replace) and with the INI_SCANNER_RAW. So basically the issue is the trash coming from Tx. Not sure what to do about it.

avatar zero-24
zero-24 - comment - 18 Apr 2017

as we need to care about B/C and you are not the only one that uses Tx i think we should revert that smal part and deprecate that behavior for 4.0

avatar mbabker
mbabker - comment - 18 Apr 2017

I always manually went through my files after a tx pull and removed the crap. I don't think there's anything for core to do with that.

avatar brianteeman
brianteeman - comment - 18 Apr 2017

Thanks for confirming its not a core issue.

avatar brianteeman brianteeman - change - 18 Apr 2017
Status New Closed
Closed_Date 0000-00-00 00:00:00 2017-04-18 14:26:02
Closed_By brianteeman
avatar brianteeman brianteeman - close - 18 Apr 2017
avatar zero-24
zero-24 - comment - 18 Apr 2017

I always manually went through my files after a tx pull and removed the crap.

do you think every 3rdparty dev is doing that? Are we alloed to break this extensions in 3.x?

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

I am not sure if everyone would have such problem. I just went through our lang files with Sigrid. We are not using QQ in our en-GB files but escaping quotes: \ " %s \ " and TransiFex makes out of it \ \ " QQ " %s \ \ " QQ "

w/o blanks. AFAIK escaping quotes signs is valid in INI file (for PHP at least). It could be indeed that other 3rd party devs are going to have similar issues. Not sure

avatar zero-24
zero-24 - comment - 18 Apr 2017

what about reveting it for now to keep b/c and remove it with 4.0? I see no issue with that.

avatar mbabker
mbabker - comment - 18 Apr 2017

It depends. As long as both the 3.6 workflow and 3.7 (basically the difference between manually parsing a file contents then using parse_ini_string() and just using parse_ini_file()) allow the same values in a string (which should be validated against our debug code in JLanguage), there is no B/C break. But if there is such a core behavioral change between parse_ini_string() and parse_ini_file() then yes it must be reverted. To be honest, it looks like the change just makes the parser more strict, the screenshot shows what really looks to be a questionable INI string (which looks to be confirmed by the command line statement posted above it).

avatar zero-24
zero-24 - comment - 18 Apr 2017

Yes my only point is this comes out per default of TransiFex and IIRC a lot of extensions use TransiFex for translations

avatar mbabker
mbabker - comment - 18 Apr 2017

But, the Transifex files as dumped are not valid INI files as they fail PHP's parser. It only passed previously because the file contents were being manipulated by us in our code. So I do not see this as a B/C break as we were allowing files that were not proper INI format to be read and now the parser is more strict to the INI format structure (at least as PHP uses it).

This is an issue with Transifex that they must address.

avatar mbabker
mbabker - comment - 18 Apr 2017

It's honestly not much different than the Registry JSON formatter being more strict about throwing errors with broken JSON strings versus blissfully returning a null value and devs/users thinking things were working with their expected data.

avatar brianteeman
brianteeman - comment - 18 Apr 2017

you could put a note in the docs i guess if you think it will effect others

avatar zero-24
zero-24 - comment - 18 Apr 2017

sure. but don't b/c means something like works with (nearly) no code change in the next version? I'm not affected and don't care but don't we try to not be the one that are blamed for any issue on 3rd party extensions.

avatar mbabker
mbabker - comment - 18 Apr 2017

It depends on your interpretation of B/C. If the resource was previously obviously broken (as can be confirmed by Radek's inline script) and we make a change that stops accepting a broken resource, to me that's not a B/C break.

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

I agree with Michael. It's rather Tx issue and ours as we didn't realised it's producing invalid language file. If we could a note it should be enough. I wouldn't say it's a B/C issue. It was broken before but worked so we couldn't see the bug.

avatar zero-24
zero-24 - comment - 18 Apr 2017

Than this is going to be a we have fixed this bug. Why did you not tested your extension on RC releases.

avatar mbabker
mbabker - comment - 18 Apr 2017

Definitely should be noted in the FAQs.

avatar zero-24
zero-24 - comment - 18 Apr 2017

https://docs.joomla.org/J3.x:3rd_Party_language_files_are_not_loaded please check and correct any issues if we have one.

avatar mbabker
mbabker - comment - 18 Apr 2017

I would not call it "third party language files not loaded", basically sounds like a big "we broke your stuff and we don't care". "Changes in language file loading" is more human friendly.

avatar Radek-Suski
Radek-Suski - comment - 18 Apr 2017

Thanks you @Sandra97

avatar Hils
Hils - comment - 27 Apr 2017

This is an issue with Transifex that they must address.

Has anyone alerted Transifex to this thread please?

avatar Gitjk
Gitjk - comment - 27 Apr 2017

@Radek-Suski
Just an additional comment - If you would use single quotes like '%s' "instead of \ " %s \ " , you won't have this problem with Transifex. I think html entities do work, too.
But even if the source language files use single quotes, some translators used "_QQ_" within their translated strings. Transifex translates these into"_QQ_"_QQ_"_QQ_", which is not tolerated anymore and results into parsing errors with Joomla 3.7.0.

There are quite a few extensions with this kind of problem now. I wonder if it would make sense to replace all the "_QQ_" in Joomla language files with single quotes or html entities for double quotes. If I remember correctly, the "_QQ_" was a Joomla 1.5 specific invention to escape double quotes and is deprecated since a long time already. If I'm not mistaken, that would not create a backward compatility issue


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/15378.

avatar CoalaWeb
CoalaWeb - comment - 13 May 2017

For the Joomla 3.7 release I updated all my extension language files from "_QQ_" to \" which as mentioned appear as\"NOT\" in the Transifex editor window but when downloaded become \\"_QQ_"NOT\\"_QQ_" To fix this in my language extension that downloads and creates my extension language packs (Similar to CTransifex) I did a string replace just before the data is written to file like this:

$file['data'] = str_replace('\\\\"_QQ_"', '\\"', $file['data']);

Now my language packs are now correctly being loaded.

For CTransifex uses you can do a similar change by adding the line above just before line 78 of:

administrator -> components -> com_ctransifex -> helpers -> package.php

I hope that helps any devs out there and I will also reach out to the dev of CTransifex.

Add a Comment

Login with GitHub to post a comment