[#32753] - JSON data stored in the database in non ASCII languages is too long.
- New
- Medium
- Build: staging
- # 32753
Is your feature request related to a problem? Please describe.
I have a module, users put into a subform field type some text in unicode. As the data is stored in JSON format (module params), its size is increasing by three times, because a json_encode function is escaping all unicode characters. My 23 KB file is becoming more than 64 KB and is not fit into limit MEDIUMTEXT. I see it's fixed in Joomla! 4, however the data is still too long.
Describe the solution you'd like
We can reduce the database size by providing an option that allows run json_encode with a flag JSON_UNESCAPED_UNICODE. If it's related to unpredictable behaviour when using globally, I suggest adding an attribute to a field, or a hook that will run json_encode with a flag JSON_UNESCAPED_UNICODE.
Additional context
Actually, I cannot see any possible issues if running json_encode in unicode mode globally. If you give some examples, when it fails, it will be sensible. I think the only reason not to use JSON_UNESCAPED_UNICODE is breaking BC with PHP5.3 that is not supported such a mode. But Joomla! 4 is not designed to cover PHP5, so it can be turned on.
joomla-cms-bot
- | Labels |
Added:
?
|
||
I cannot see any possible issues if running json_encode in unicode mode globally
if database table will have non UTF8 collation, it will explode.
And/Or while table export/import by some tool that may break non escaped UTF symbols.
This is case when better safe than sorry.
@Fedik all Joomla! tables in utf8mb4 encoding.
Not in all scenarios. There is one valid scenario which is the reason why we still have the utf8mb4 conversion in J4. If someone is on J3.10 and migrates his or her database from an old MySQL or MariaDB server which doesn't support utf8mb4 to a newer one which does, then he or she doesn't check in "Extensions -> Manage -> Database" that there are database problems (missing utf8mb4 conversion) and updates to J4. In this case the conversion from utf8 to utf8mb4 is performed by the update to J4, so after the update it is true what you say, but not before and not during the update, and during the update the new J4 PHP code is already used when running the database updates.
In general with database things we have to keep in mind the updating process from older versions. We can't just assume everybody runs a newly installed J4 where things may be fine, we also have to consider scenarios like the above.
Hackwar
- | Labels |
Added:
No Code Attached Yet
bug
Removed: ? |
||
If I remember correctly then JSON_UNESCAPED_UNICODE will not work with utf8