[#9688] - Use the phputf8 library to strip invalid UTF-8 characters
- Closed
- 13 Aug 2016
- Medium
- Build: staging
- # 9688
- Diff
- mbabker:Indexer-UTF8-Check
- Success continuous-integration/travis-ci/pr The Travis CI build passed Details
User tests: Successful: Unsuccessful:
Summary of Changes
In the case of the OP on #9686 he is getting errors because of the use of iconv() however it's possible for it to not be installed. This changes the stripping of invalid UTF-8 characters to use the phputf8 library instead (note I honestly don't know if there's a difference in the result from iconv() versus utf8_bad_strip() so someone may want to test that).
Testing Instructions
Smart Search indexer should still work and correctly strip invalid UTF-8 characters.
| Status | New | ⇒ | Pending |
joomla-cms-bot
- | Labels |
Added:
?
|
||
| Category | ⇒ | Libraries |
I thought iconv was a requirement - i know it used to be
Its availability is tested in com_admin's sysinfo view but AFAIK it isn't documented anywhere as a requirement.
The only places iconv() is explicitly used in Joomla's API is here and StringHelper::transcode(). SimplePie and the IDNA Convert libraries both will optionally use it if available in a couple of functions but those are also wrapped in function_exists() checks and try to use the mbstring extension if available otherwise. And like I noted its support is tested in the system info view but that's really it.
I would expect iconv to be considerably faster than something written in PHP, although that is a hypothesis that should be tested. If that is correct then we should prefer iconv and only fall back on something else if it isn't available.
After the test by the original poster, I also suggest we check if iconv exists and fall back to utf8_bad_strip if not.
| Category | Libraries | ⇒ | Administration Components |
| Status | Pending | ⇒ | Closed |
| Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2016-08-13 19:01:59 |
| Closed_By | ⇒ | mbabker |
could we first check if iconv is installed?