This works also.

@wilsonge Please chose which one you like more, this one or #28587 .

Testing Instructions

Thanks to @infograf768 for the testing instructions. I've added the update test to make sure I haven't made a mistake in the update sql script.

Test 1: New installation

Step 1: Patch and make a clean install.

Step 2: Create and publish an article which contains

Chinese: 不能创建文件

Greek: Εγκατάσταση Γλωσσών

German: Europäer

French: être noël

Simple chinese: 不

Four bytes character:
?
or
?
equivalent of U+20E9D  ?
groupés par 3 不??创

Step 3: Create a smartsearch module in frontend.

Step 4: In frontend, search for any Chinese character or group of characters, but specially for the 4 bytes
?
and
?

Result: OK and hightlighting is correct.

Test 2: Update

Step 1: Update a 3.10 to 4.0-dev plus the patch of this PR applied, using the update package built for this PR or the corresponding custom update URL. Packages and update URL can be found here: https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/28592/downloads/31062/.

Step 2: Repeat steps 2 to 4 of the previous "Test 1: New installation".

Result: Same as for "Test 1: New installation".

I would prefer this over #28587.

I have tested this item ✅ successfully on 0b20612

with mysql 8.0.19

I have tested this item ✅ successfully on 0b20612

Will close the other one

RTC

@alikon @infograf768 I hope you have also done "Test 2: Update".

my bad and laziness no.... i'll do sorry

Update test is important to check that I haven't made a mistake in the update SQL and that the order of processing is correct, i.e. at the end collations are like they should be for the tables and columns handled by this PR.

@alikon @infograf768 wait with the update test. I've just updated to latest 4.0-dev and so new update package will be built by drone.

New update package and custom URL have been built by drone. I've updated the link in the testing instructions.

@alikon Ready for the update test now.

@infograf768 If you have done both tests before, installation and update, please just mark your test result again. Otherwise, if you haven't done the update test: Could you do that test, too? It is important to check there is no mistake in the update sql.

test done and can confirm that work updating from 3.10 to 4

I have tested this item ✅ successfully on 938edd1

this time tested the update too

Not sure if related but Joomla 3x table fields such as images in #__content, params in #__modules and #__menu are converting unicode characters to 6-bytes. They appear unicode in the backend but you see multi-byte in db.

@humblehumanbeing

Not sure if related but Joomla 3x table fields such as images in #__content, params in #__modules and #__menu are converting unicode characters to 6-bytes. They appear unicode in the backend but you see multi-byte in db.

Looks like you are confusing bytes and bits. Unicode UTF8 is max 4 bytes.
Please give some examples of what you mean.
For params, getting the format \u....\u... is totally normal and is unrelated to the issue here.

Sorry if it is the case.
In the backend create an article, enter 'ş İ Ğ Ö Ç' for fulltext image alt tag.
Check the field images via phpmyadmin
It is encoded as '\u015f \u0130 \u011e \u00d6 \u00c7'
Same in module/menu params.
This makes db manipulation, find and replace for example, via a db software nearly impossible

These are JSON encoded and it is not related to the issue here with finder.
See https://www.php.net/manual/fr/function.json-encode.php

Thanks for the clarification.

Thanks!