OK it's kinda hard to test but with this PR what happens when I upgrade from say 2.5 to a 3.5.2 directly?

Same as without this PR: It will run the conversion once, because a 2.5. never was converted.

Or did you mean 3.5?

I did mean 2.5 - but thinking about it this morning you are correct. I should try and think less at 1am i think :D

If you absolutely want to keep that subquery which I remove with this PR,(which was a kind of snake oil I think meanwhile) I could use the 1st 5 chars of the schema version to find out if I am just updating a 3.5.0 or not. Problem was: Extension version is updated before the schema update sql scripts run, and I have no version_compare in SQL, so I cannot check for the post-update version (can be a 3.5.1 or a 3.5.2 or a 3.5.3 ...), I have to check for the pre-update version (3.5.0) to see if I have to run again. On updating, only those schema sql are executed which have schema in their file name larger than pre-update schema, so when updating from a pre 3.5.1, the statement will reset the conversion status, too, which does not matter because there we have to convert anyway. On a 3.5.1 or later it will not run because the SQL is a 3.5.1*sql, and if db schema is up to date, it will not be run anymore by the installer/updater. Only db fixer will check that script, but he will ignore "update" statements. So it will work as desired with and without the snakeoil, but if you want some of that oil I maybe can use 1st 5 chars of schema version and not the extension version in that subquery which I removed with this PR here.

Let me know please as soon as you can if you want some snake oil subquery with schema table and schema version, then I will prepare this, or if it is OK like it is now.

With snakeoil it would be:

UPDATE #__utf8_conversion SET converted = 0 WHERE (SELECT COUNT(*) FROM #__schemas WHERE extension_id=700 AND version_id LIKE '3.5.0%') = 1;

It would not change anything in practice but help the reader to understand maybe, and would make it safe for the case if someone runs the script again.

Let me know if you want this, or if I shall leave this PR like it is now, without the snakeoil subquery,

@wilsonge Another idea: We remove this new schema update sql and create a new init_check.sql or so in the folder for the conversion scripts, having this update statement (including the subquery as mentioned above with schema version instead of extension version) and run this always before checking if conversion is necessary in script.php and in the schema manager's populateState (or the changeset's constructor). Then depending on the subquery (is no snakeoil anymore then, has real purpose then) the conversion status is reset or not. This would work for both upgrade methods. Also this would be a temporary solution until we have found the real way do handle both core and extensions the same way, but I could have it ready soon, so if you would say it would be good to have it in 3.5.1 I could have it ready in time.

@wilsonge Hmm, I just see script.php would come too late ... when it runs, the schema is already updated ... so my previous idea would be only for the schema manager ... for normal update a schema update sql like now in this PR would be ok. I only would have to make schema manager run the query, too (with correct subquery) before doing the checks. I will see if I can find an easy way for that.

I don't want to keep the subquery if it's not required :) I just got an edge case scenario in my head last night - but I don't think it's actually valid - this is fine :)

Solution for all is on the way, wait my next commit. I tested here and it works. Will update test update container behind custom url and testing instructions so you also can use a clean 3.5.0 from new install.

@wilsonge That's it now. Have to update testing instructions and so on now.

@wilsonge This is ready for test now. Trust me, I tested here.

@andrepereiradasilva Sorry, but if you can and want: One more hotfix to be tested.

Sorry Richard i only have php 7 now, so can only test with 3.5.0+ Joomla versions.

No problem, all required tests are 3.5.0. You have to patch it as described in the prerequisites and save it in a dump then for multiple reuse. Thanks in advance for testing.

I have tested this item successfully on 6e34f2e

Test 1 and Test 2 with success.

Didn't do any "Optional free tests".

@andrepereiradasilva Thanks for testing.

Just for curiousity: You could update with my custom URL from 3.5.1-rc2 unpatched to 3.5.1-rc2 patched during Test 1?

I thought it will not work because already being on the same Joomla! version as the offered update, that's why I just changed testing instructions a bit so Step 3 includes also fallback of the filesystem (but a patch of the version in manifest cache and so could also have been sufficient).

Or did it not work but you helped yourself?

Just for curiousity: You could update with my custom URL from 3.5.1-rc2 unpatched to 3.5.1-rc2 patched during Test 1?

what step are you referring?

@andrepereiradasilva Ah, sorry, yes, I was paranoid. Restoring the old db is sufficient. It also includes the manifest cache in the db, the one in the XML file is not relevant for version check. So was false alarm. I will change testing instructions back now to they are as easy as before.

I worried that when doing Step 4 of Test 1, updating to my patched 3.5.1-rc2 with file system being already 3.5.1-rc2 from the steps before, the update component could report that there is no update because the installed version is already up to date (3.5.1-rc2). But that is not the case because the restored 3.5.0 database also includes the manifest cache. So all ok, test instructions changed back to successfully tested version.

ok

@wilsonge This time it works, would be great if you could test and if OK use for 3.5.1. The subquery now is not snake oil anymore, it is needed to make the reset of conversion status not happen on every creation of the changeset.

I don't have time. I have a big work deployment tomorrow that I need to bug fix tonight :( I'll try and find someone to test this for you!

Oh, that would be nice, if you could find someone. And maybe it is not so urgent, because if you have no time for testing, you also have no time to release 3.5.1 .Good luck with your work.

@Webdongle I need some more tester(s) for this PR. Do you think you can find time and mood for that?

Testing now

@Webdongle Can you repeat Test 1 Step 3 and in addition fall back also to the old 3.5.0 filesystem (hope you saved that), and then continue? If that works, I have to adapt testing instructions to that.

@Webdongle Or maybe it will be just enough to use the "Clear cache" button on the Joomla! Update component page? So or so it is not a problem caused by this PR, I only thought I save some work when falling back with database only and not with all to test again updating the 3.5.0 to the patched 3.5.1-rc.

So please repeat Step 3 of test 1 and then check if the "Clear cache" button on the Joomla! Update component page works, and if not, also fall back with file system, and then Step 4 should work.

Sorry for the inconvenience.

@Webdongle Or maybe it need a page reload of the updater page in Step 3 after playing back the 3.5.0 database, so the updater knows it is a 3.5.0 now? Or did you maybe play back the wrong database dump, a 3.5.1-rc2 and not a 3.5.0?

@Webdongle I see it was missing in the testing instructions. I have corrected the instructions: Added backup of file system to preparation, and added falling back to backuped file system in Step 3 of Test 1.

Can you test again?

The dump I imported was the 3.5.0 ... done it twice (separate installs separate folders/databases) to make sure
Cleared cache even used another browser
No I didn't keep the files

Test 1 step 3 I can complete by installing a fresh 3.4.8 and updating to 3.5.0 then update via http://test5.richard-fath.de/list_test5.xml . But the method of test 2 is in question because it also has a mismatch of files to database. Because imho using older database tables with newer files would not be a sound test.

Either the test instructions are illogical or I am interpreting them incorrectly. In either case I am not comfortable with my ability to accurately interpret my results as a success

@Webdongle I had just updated the testing instructions to fall back both database and filesystem to 3.5.0 in Step 3 of Test 1, so this should work now.

Test 2, new files with old database and then using discover extensions and database fixer is a valid update method, see documentation here: https://docs.joomla.org/J3.x:Upgrading_from_Joomla_3.4.x_to_3.5, Method C. And exactly this is what Test 2 is testing. So please check and think over.

If you still not feel comfortable after reading, then mark it as not tested maybe, but not as failed.

OK got it now ... suggest

Install 3.4.8
Update to 3.5.0
Copy the updated site
Test on the copy
Make another copy of the site do next tests on that copy etc ?

I have not tested this item.

Install 3.4.8
Update to 3.5.0

or

Use a new 3.5.0 and patch with SQL statement so it has one of the schema differences as it would have after updating from 3.4.8.

are the 2 alternative starting conditions mentioned with the "Pre-requisites" section, maybe I described that too complicated.

The "Copy the updated site (both database and files)" comes then with section "Test preparation".

And then the test can be done as described, with falling back both database and files to 3.5.0 in Step 3 of Test 1.

Sorry when I wrote in a too complcated way maybe.

Yep it is complicated but I got it now. Deleting and replacing is long complicated. Making copies from the original is simpler

Test 2
Matching the 3.5.1-rc2 patched files to the 3.5.0 database tables I get

Error displaying the error page: Application Instantiation Error: Table 'j350c.#__session' doesn't exist SQL=DELETE FROM #__session WHERE time < '1459456927'

Hmm, looks unrelated to this PR to me. I did not have that when testing. What looks strange to be is your table seems not to have the right prefix? 'j350c.#__session' ... should that not be 'j350c_session'? And in the SQL, shouldn't there also be the table prefix be replaced? Something wrong with your db dump?

Normally that means that the database is missing. Dealing with the session is literally the first thing that happens

So or so it is not related to my PR.

Tried a different way

Installed fresh 3.5.0 (idx_name size 100)
overwrote the files (except configuration.php)
Extensions >>> database showed 4 errors
'Fix' worked
idx_name size remained 100

Installed fresh 3.4.8 (idx_name size blank)
Updated via Joomla update component to 3.5.0 (idx_name size 191)
overwrote the files (except configuration.php)
Extensions >>> database showed 4 errors
'Fix' worked
idx_name size changed to 100

This looks like a success to me. The error before must have been browser cache

It everyone is happy I will mark it as a Successful test ?

Yes, looks like a successful test to me. I would not fight you marking it so

I have tested this item successfully on 6e34f2e

After several tests with various combinations this patch passes Full Success

@andrepereiradasilva @Webdongle Thanks a lot for testing. Sorry for maybe too complicated instructions.

@wilsonge The changes for schema manager changeset detecting the special update query to reset conversion status is a quick solution for 3.5.1, but it is not a dirty hack which should be removed later.

It can be kept even after a future solution for handling conversion scripts for core and extensions, as an additional method to reset conversion status.

About what's the best way for such a future solution (I try to avoid the word "final solution" because this has a special meaning from bad times of German history) I would like to talk with you as soon as you have mood and time, so I not go a wrong way.

RTC

@wilsonge Should this here have milestone 3.5.1? I think so because without it, the conversion will not run again when updating a 3.5.0. And when shall we talk about the remaining stuff to be done for utf8mb4 conversion for extensions?

Merged with 3716365 (GitHub went "down" and didn't automatically close PR)

Hmm, I hope it was not my PR which made GitHub go down

No github has general issues these days :)

Hmm, just after the Panama files were published, GitHub has issues ... maybe there's a relation? (joke)