Let me know when you are ready for testing, I have both kinds of databases, i.e. with and without utf8mb4 support. Btw, if you find time ... there is still one easy test I need for my other PR.

I'm not going to have time tonight (been working on this and talking to marketing for 3.5) but I have cloned it onto my laptop for the train ride in tomorrow :) Generating and doing some testing of the demo package now :)

Another thing I saw in your test description: I think you mean update stream "Custom URL", not "Testing", right?

No I mean testing :) Because of this https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_joomlaupdate/models/default.php#L247-L254

Ah, and this works? Good to know.

well theoretically - i've never tried it before so it could be a disastrous failure :P

Ok, have a good working now, and sorry for disturbing. Am available for testing tomorrow.

No it's fine - thanks for picking up on that mistake :)

I think I'm going to loose the will to live soon. Tonight is one of those nights where I got as far as tagging a beta4 (locally) and going through the full release strategy on my pc and extension manager tells me it's invalid -.-

I'm not sure if this is going to work or not - but let's try opening this to the floor for testing :/

Trying to install using "testing" and http://gwilson.org/Joomla_3.5.0-beta3-Beta-Update_Package.zip just does not work here. I have a lot of errors. Too many to list here.

Therefore I tried using Extension Installer:
I get:

Then going to Database, I get:

I had to use the Fix button to solve the remaining issue.
I guess therefore that my test is unsuccessful.

Tried to update using the Joomla Updater, put beta3 into tmp folder, set update to testing.
Update failed.

Not sure if the updater is trying to use the file in TMP, doesn't look like it?

Tried the extension route. Got an SQL error same as @infograf768 and same error in database ui.

System did update but it decided to update to beta4, not beta3...frisky updater!
Database encoding looks the same as before.

@wilsonge Sorry, did not read your skype last night. the answer to your question is related to what JM had, the empty SQL error at the top. We have to trim the SQL after having removed comments becuase otherwise query is not detected to be empty (space only) and is executed.

Let me know when fixed, I'm ready to go again.

@wilsonge I've meanwhile answered your question on skype. I will later test this too and see what I can find out.

Gary you're hitting the SSL certificate issue :( that error is unrelated to this PR - But JM's test is good enough I'll go back through things again!

@wilsonge Check your skype ;-)

It updating to beta 4 is expected. I created it as a beta 4 test package and renamed the zip to exploit a feature in the Joomla Update component :p anyhow clearly an issue so I'll set to work on it!

In order to get this issue here solved in the right way, my PR #9303 should be tested and if ok be merged first.

Then this PR here could be changed so the installer script.php can check if conversion status in database is equal to excpected conversion status according to utf8mb4 support (2 is yes, 1 if no) at script runtime, and run the conversion only if those values are not equal. This would also handle correctly data having been migrated from old non-utf8mb4 to new utf8mb4 database.

In the same way I will have to change with a new PR the schema manager (db fix) database.php so it runs the convertion only if necessary.

So whoever has time please test PR #9303 even if it does not have the 3.5.0-blocker label (yet).

OK I've just rebased this on staging. So this branch now includes @richard67 's changes

@wilsonge Well, the only thing which is missing in this PR here to make it perfect is a check if the conversion status in database (column "converted" in table "#__utf8_conversion") is equal to what is expected according to utf8mb4 support at script runtime, i.e. variable $converted (setting of this vasriable has to be moved to before step 1 of conversion).

If you want I make a PR to your repo and branch of this PR.

The conversion then should be run only if expected status and saved status in db differ.

@wilsonge Ah or maybe I should make a PR for having it in the database driver as a function, like this serverClaimsUtf8mb4Supoort (or so), which returns true for all other server types and in case of mysql true if conversion status in db is 2 (=utf8mb4) and false otherwise? You could use this then in your PR here. Or you do it with your PR here, too? Let me know what you prefer pls.

Anything in JDatabaseDriver I can't use in here for the same reason as the other functions :( - if you are updating through extension manager you are using the old version of JDatabaseDriver in this function

Ahhh sorry, I forgot (must be age related)

Then the only place where we would call this function would be the input filter and maybe the database fixer, but for the latter it would not really be neccessary because can check that himself.

Await my PR to your PR here with necessary changes for the check.

Long term putting it in the database check is ok - because I think in the not too distant future (for various reasons) we will stop people using Extension Manager to upgrade Joomla - maybe as soon as 3.6 -- so it can't hurt to have it there - but we can't use it in this PR for now

@wilsonge Have sent PR no. 23 wilsonge#23 to your repo. Please check also my comments.

So is this ready to test GDubya?

Not right now. I'm just having a think about a few things!

OK I'm happy with this now - i'm just going to go generate a new testing zip

@wilsonge Please merge my last commit to your repo and this branch first ... I had to correct errors from my previous ones.

Joomla! Update component with provided custom URL works well for both types of database with and without utf8mb4 support. Will check Extensions Installer now.

@Bodge-IT ready for testing now!

I have tested this item successfully on b2c8a55

Tested both with success database with and without utf8mb4 support.

Could not test extension installer because or remote test environment and upload problems.

Hint for other testers: The custom update URL (xml) works only for updating 3.x but not for 2.5 because the details XML referred in the 1st xml is not prepared for that. With the zip this should work also for 2.5.

I suggest to add the link to the Database page in the language string, i.e.
JLIB_DATABASE_ERROR_DATABASE_UPGRADE_FAILED="MySQL Database Upgrade failed. Please check the <a href="index.php?option=com_installer&view=database">Database Fixer</a>."

I have tested this item successfully on b2c8a55

Updated using Custom URL:
Database info shows OK, tables in DB show utf8mb4_unicode_ci

Just tested this on another site using the extension method and update failed.

Pressing Fix finished the update

I have tested this item unsuccessfully on b2c8a55

Update method: Install from URL

Upgrade path: 2.5.28 (clean with sample data) -> 3.5.0-beta3 (your custom zip)
Result: HTTP 500 error

Upgrade path: 3.2.7 (clean with sample test data) -> 3.5.0-beta3 (your custom zip)
Result: HTTP 500 error

Upgrade path: 3.4.8 (clean with sample test data) -> 3.5.0-beta3 (your custom zip)
Result: upgrade ok, but tables not converted to utf8mb4_unicode_ci (converted = 0)

I have tested this item unsuccessfully on b2c8a55

Meanwhile I could fix my upload problems and so could test extension installer, too, and can fully confirm @andrepereiradasilva 's findings.

Also the 500 error in some case.

It seems we have a problem only whn using extension installer, update component works.

@wilsonge Maybe you rename this pr to solve the update component method only and say it's ok and do the extension installer with another new PR?

when another test is needed say so

The problem is the 500 error. Can someone confirm if we have a 500 error without this PR applied when using Extension Manager? I'd be happy to merge this if it's confirmed that this isn't the source of that?

Well, but even then we have to solve that with extension manager the conversion is not performed at the end.

In a worse case I'm happy to use the same message to ensure that users are shown a message to go to the database fixer tabs. The majority of people use Joomla Update! so to me that's the most important thing. If Extension Manager people don't get a 500 error and show a "kind" error message to go check Database Fixer manually I'm happy with that for 3.5.0

Can check that tomorrow, if I get 500 error also without this PR.

@wilsonge
Update method: Upload

Upgrade path: 2.5.28 (clean with sample data) -> 3.5.0 beta 3 (official)
Result: Upgraded with success

So, the 500 error occurs only with this changes (it seems)

We've made quite a few changes since Beta 3 - I wouldn't call that a definite result - but it's enough for me to have a long look into this and try and figure out wtf is going on. Thanks @andrepereiradasilva :)

Or with other changes in staging since Beta 3 ... would be good to test this with staging. I will find some time for that tomorrow afternoon maybe.

if anyone can make an install package of the latest staging i can test.

Give me an hour and I can have one

Use Beta 4 https://github.com/joomla/joomla-cms/releases/tag/3.5.0-beta4

@wilsonge
Update method: Upload

Upgrade path: 2.5.28 (clean with sample data) -> 3.5.0 beta 4 update package (official)
Result: 500 error

error log

PHP message: PHP Notice:  Undefined property: InstallerController::$input in /path/to/joomla-25/administrator/components/com_installer/controller.php on line 37
PHP message: PHP Fatal error:  Call to a member function get() on null in /path/to/joomla-25/administrator/components/com_installer/controller.php on line 37

OK that can't be new - that's a standard JInput line - and as JInput wasn't available in Joomla 2.5 I guess it kinda makes sense (although it would mean that any 2.5 to 3.x install would do that - depending on when we swapped that instance from JRequest to JInput). What about 3.4.8 or 3.2?

please keep in mind that in my test after the 500 error i can login back to a broken admin panel, and i'm at 3.5.0 beta 4, so this error log exists everytime i open an extensions view, or anything that uses the filter. so the problem seems to happen in the upgrade but i have no information of what problem.
The error log (500 error) happens after the upgrade process i think (when reloading the install page)

@wilsonge
Update method: Upload

Upgrade path: 3.2.7 (clean with sample data) -> 3.5.0 beta 4 FULL package (official)
Result: 500 error, but after login again i'm on 3.5.0.beta4 and all seems to work...

error log

PHP Fatal error:  Class 'Joomla\Registry\Registry' not found in /path/to/joomla-327/libraries/cms/module/helper.php on line 169

Fml. What a mess. We only put in JRegistry in 3.3 (iirc) so again I guess it makes sense - it's trying to load old versions of files....

@wilsonge
Update method: Upload

Upgrade path: 3.4.8 (clean with sample data) -> 3.5.0 beta 4 UPDATE package (official)
Result: Success

it was something that changed from beta 3 to beta 4, since beta 3 worked fine.

What worked upgrading to beta 3 sorry?

everything. upgrade from 2.5.28, from 3.2.7 and from 3.4.8 to 3.5.0 beta 3 all worked fine.

ups i got a 500 error now .... upgrading from 2.5.28 to 3.5.0 beta 3

JInput wasn't available in Joomla 2.5

JInput was in 2.5. JController::$input didn't exist in 2.5 (JControllerLegacy::$input in 3.x).

The class aliasing is new as of Joomla 3.3.0.

If this is all updating via the Extension Manager it makes sense why you'd hit these errors (because everything pre-3.5 is still loaded into memory). Why it's new with the last package beats the piss out of me but all roads point to that upgrade method causing catastrophic failures across the board.

yes, it seems it happens with 3.5.0 beta 3 too

just tested from 2.5.28 to 3.4.8 update package and worked fine, and using the same method (extension -> install -> upload)
i have been logged out as normal after 3.4.8 update

I just tested going from 3.4.8 to 3.5 beta 4 succesfully - clearly it's not a reliable 500 error :P But as it is the same (seemingly) without and without this patch I'm going to merge this and then we can go back and work on extension manager

I just tested going from 3.4.8 to 3.5 beta 4 succesfully - clearly it's not a reliable 500 error :P But as it is the same (seemingly) without and without this patch I'm going to merge this and then we can go back and work on extension manager

I never got problem from 3.4.8 to any beta.
The problems are in older versions (2.5.28 and 3.2.7) to 3.5.0 beta.

But the problem has nothing to do with this PR. So yes this should be merged.

@wilsonge What about adding missing query downgrades to libraries/cms/installer/installer.php so future update sql scripts can create tables in the right way, and changing back the administrator/components/com_admin/sql/updates/mysql/3.5.0-2016-02-26.sql also (to use utf8mb4 which will be downgraded if necessary)? Would make sense to do that, or not?

@wilsonge @richard67 is there any way for the extension manager recognize that you are updating joomla and when it does so it uses the normal upgrade process.

This way we wouldn't need to be always code and test several different upgrade processes. Only one.
I don't know if this is possible, but if so, i think it would be more easy for everyone.

@andrepereiradasilva That was also my question, and I don't have an answer yet.

No, it's not. I commented this elsewhere (don't remember where anymore), but the first point that the API is aware of what is being installed is well after the Extension Manager has called the JInstaller library. This library shouldn't be handling this type of detection/redirect activity (the fact we have so many API level functions coupled to components means we have absolute garbage abstraction layers, see how tightly coupled com_categories and com_tags are to the core).

With that said, JInstaller::install() (which is what the Extension Manager's Install view will always trigger, the internal API will figure out whether it needs to treat it as an update to an existing extension) dispatches an onExtensionBeforeInstall event. Given there is already a PlgExtensionJoomla you could theoretically add a listener for that event to the plugin which handles the redirect.

And that plugin is pretty much guaranteed to always be enabled. Otherwise no extension ever gets its update sites registered (seriously, an "integral" part of extensions is left to the plugin layer instead of integrated directly into the install API, I'm still fixing my desk from when I head -> desk after discovering that).

And that plugin is pretty much guaranteed to always be enabled. Otherwise no extension ever gets its update sites registered (seriously, an "integral" part of extensions is left to the plugin layer instead of integrated directly into the install API, I'm still fixing my desk from when I head -> desk after discovering that).

Lol. always thought that plugin was very very very strange too!

@mbabker @wilsonge @richard67
just an heads up, it seems #9331 was the cause i the 500 errors.

Using a patched version with the sql error corrected (just a space after --) i was able to upgrade from 2.5.28 to 3.5.0 beta 5 (patched version) without any problem.

One thing i notice was that in that upgrade the tables weren't converted to utf8mb4_unicode_ci, the rest worked fine.

Anyway i will wait for @richard67 PR to do the (i hope) final tests.

@andrepereiradasilva Please check #9332 .