? Pending

User tests: Successful: Unsuccessful:

avatar richard67
richard67
22 Sep 2019

Pull Request for Issue #24535 (part).

Summary of Changes

Continuation of Pull Request (PR) #24675 .

This PR fixes all datetime columns of the com_contacts database table #__contact_details so there will not be any Invalid value '0000-00-00 00:00:00' for datetime error anymore on MySQL 5.7 or later when strict mode is enabled.

Currently this table only contains 2 not nullable datetime columns, createdand modified. Those will be handled like the same columns of the #__banners table in PR #26372 , i.e. they will get no default value.

The default value is only used when inserting new records without specifying values for that particular column. Not having a default will enforce to insert new records with values for these columns being provided and throw an SQL error if some of these values is not specified, i.e. such errors will not be hidden anymore.

Old data will be updated as little as possible. The created column will be not touched at all. We can assume that for core components there is no data with values '0000-00-00 00:00:00' on MySQL or '1970-01-01 00:00:00' on PostgreSQL, and data created by 3rd party components should not be modified. The modified column will be set to the value of the created column only if it has value '0000-00-00 00:00:00' on MySQL or '1970-01-01 00:00:00' on PostgreSQL. Our PHP already sets the modified time to the created time when saving new records, i.e. modified = created means never modified.

Note on updating old 4.0 sql update scripts

Since we are not in Beta phase yet and so don't have to support updates from 4.0-Alpha-x to 4.0-Alpha-y or between nightly builds, we can change the existing 4.0 update scripts (but of course not pre-4.0 scripts). Later when in beta this will not be allowed anymore because we have to support updating from Beta-x to Beta-y (with y > x of course), so now is a good time to fix them.

Furthermore it makes sense to keep the schema updates for the datetime columns of each table together in one script, because on MySQL it might be necessary to combine all changes for a particula table into 1 single ALTER TABLE statement in order to run properly in strict mode (i.e. strict tables enabled). Currently the db checker/fixer can't understand such statements, but it might be necessary to teach that tool soon to do that.

That's why this PR updates the existing sql update script and doesn't create a new one.

Testing Instructions

Testers please report back the database kind (MySQL or PostgreSQL) on which you have tested.

If you have both MySQL and PostgreSQL, please test on both if possible.

Test 1: New installation

  1. Apply the patch on a clean 4.0-dev branch using git or merging manually, or apply it on an installation of current 4.0-dev using patchtester.
  2. If used patchtester on an existing installation in step 1, delete file configuration.php and delete all Joomla database tables in PhpMyAdmin or PhpPgAdmin (depending on your database type).
  3. Do a new installation, login to backend, confirm the statistics dialog, go to global config and set error reporting to maximum or development in server settings.
  4. Play around with contacts, create some, publish some, ...
  5. After any action, check in your database the relevant datetime/timestamp without timezone columns.

Result: See section "Expected result" below.

Test 2: Update sql script

  1. Install a clean clean 4.0-dev, login to backend, confirm the statistics dialog, go to global config and set error reporting to maximum or development in server settings.
  2. Apply the changes from this PR e.g. manually or with patch tester.
  3. Open PhpMyAdmin or PhpPgAdmin (depending on your database type), select your database and then go to the SQL command input.
  4. On MySQL copy the first line of file installation/sql/mysql/joomla.sql into the SQL command window but don't execute the commands yet:
SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";

This switches off strict mode to the SQL will run on MySQL 5.7 or later.

  1. Copy the content of file administrator/components/com_admin/sql/updates/mysql/4.0.0-2019-04-22.sql or administrator/components/com_admin/sql/updates/postgresql/4.0.0-2019-04-22.sql (depending on your database type) into the SQL command window, in case of MySQL below the previously pasted commands, but don't execute the commands yet.
  2. Replace #__ by your database prefix in the SQL statements pasted before in the SQL input window.
  3. Put the cursor to the beginning of the 1st SQL statement in the SQL input window and now execute all SQL commands.
  4. Play around with contacts, create some, publish some,...
  5. After any action, check in your database the relevant datetime/timestamp without timezone columns.

Result: See section "Expected result" below.

Expected result

Contacts work as well as before. In database there are no columns of type datetime having value '0000-00-00 00:00:00' in a MySQL database, and there is no invalid default value anymore in MySQL >= 5.7 with strict mode on in that kind of database.

There is one exception: When checking in a contact using com_checkin, the checked_out_time is set to '0000-00-00 00:00:00' on MySQL and '1970-01-01 00:00:00' on PosgreSQL. This will be changed with a separate, future PR for com_checkin. Checking in an item with the lock icon in list display works, i.e. there checked_out_time is set to NULL.

Actual result

Contacts work. In a MySQL database there might be columns of type datetime having value '0000-00-00 00:00:00', and the default value of these database columns is invalid in MySQL >= 5.7 with strict mode on.

Documentation Changes Required

Maybe core developer docs and extension developer docs should be updated to encourage them not to use '0000-00-00 00:00:00' on MySQL anymore but use real NULL and not abuse '1970-01-01 00:00:00' on PostgreSQL as a speudo null date anymore and use real NULL values also there.

avatar richard67 richard67 - open - 22 Sep 2019
avatar richard67 richard67 - change - 22 Sep 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 22 Sep 2019
Category SQL Administration com_admin Postgresql Installation
avatar richard67 richard67 - change - 22 Sep 2019
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - change - 22 Sep 2019
Category SQL Administration com_admin Postgresql Installation SQL Administration com_admin Postgresql Front End com_contact Installation
avatar richard67 richard67 - change - 22 Sep 2019
The description was changed
avatar richard67 richard67 - edited - 22 Sep 2019
avatar richard67 richard67 - change - 22 Sep 2019
The description was changed
avatar richard67 richard67 - edited - 22 Sep 2019
avatar richard67 richard67 - change - 22 Sep 2019
The description was changed
avatar richard67 richard67 - edited - 22 Sep 2019
avatar richard67 richard67 - change - 22 Sep 2019
The description was changed
avatar richard67 richard67 - edited - 22 Sep 2019
avatar richard67 richard67 - change - 22 Sep 2019
The description was changed
avatar richard67 richard67 - edited - 22 Sep 2019
avatar richard67 richard67 - edited - 28 Sep 2019
avatar richard67 richard67 - change - 28 Sep 2019
The description was changed
avatar richard67
richard67 - comment - 28 Sep 2019

Please wait with testing until PR #26295 is merged.

avatar richard67 richard67 - change - 28 Sep 2019
Title
[4.0] [WiP] [com_contacts] Fix default value for not nullable datetime column
[4.0] [com_contacts] Fix default value for not nullable datetime column
avatar richard67 richard67 - edited - 28 Sep 2019
avatar richard67 richard67 - change - 28 Sep 2019
The description was changed
avatar richard67 richard67 - edited - 28 Sep 2019
avatar richard67 richard67 - change - 3 Oct 2019
The description was changed
avatar richard67 richard67 - edited - 3 Oct 2019
avatar richard67 richard67 - change - 5 Oct 2019
The description was changed
avatar richard67 richard67 - edited - 5 Oct 2019
avatar richard67
richard67 - comment - 6 Oct 2019

Ready for test now as PR #26295 has been merged.

avatar richard67 richard67 - change - 6 Oct 2019
The description was changed
avatar richard67 richard67 - edited - 6 Oct 2019
avatar richard67 richard67 - change - 6 Oct 2019
The description was changed
avatar richard67 richard67 - edited - 6 Oct 2019
avatar richard67 richard67 - change - 6 Oct 2019
The description was changed
avatar richard67 richard67 - edited - 6 Oct 2019
avatar alikon alikon - test_item - 16 Oct 2019 - Tested successfully
avatar alikon
alikon - comment - 16 Oct 2019

I have tested this item successfully on ec174ad

postgresql 11.5


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/26379.

avatar wilsonge wilsonge - change - 16 Oct 2019
Status Pending Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2019-10-16 19:07:36
Closed_By wilsonge
avatar wilsonge wilsonge - close - 16 Oct 2019
avatar wilsonge wilsonge - merge - 16 Oct 2019
avatar wilsonge
wilsonge - comment - 16 Oct 2019

Thanks!

avatar NickSdot
NickSdot - comment - 10 Jan 2020

Thanks for that!

May I know what you mean with this here?

We can assume that for core components there is no data with values '0000-00-00 00:00:00' on MySQL or '1970-01-01 00:00:00' on PostgreSQL, and data created by 3rd party components should not be modified.

I do ask this because a J!3 installation is full of '0000-00-00 00:00:00' values in several core components. Which is quite a hustle since an update like this will not work without changing the MySQL configuration.

UPDATE `#__table_name` 
SET `column` = NULL
WHERE `column` = '0000-00-00 00:00:00';

It simply will result in:

Incorrect datetime value: '0000-00-00 00:00:00' for column 'column' at row 1

Or for date:

Incorrect date value: '0000-00-00' for column 'column' at row 1

Do you talk about a fresh J!4 installation? If so, what about old J!3 installations which already use the not compatible values?

avatar wilsonge
wilsonge - comment - 10 Jan 2020

We have migrated all of J4 over to use real null values for null date time's due to the mysql 8 move to (the sql standard) null date time of 1001-01-01. obviously we can't just move to that without breaking mysql 5.6 and below.

J3 values in core components will get upgraded as part of the standard sql upgrade process. And it's going to be documented for 3rd party extensions who integrate with this

avatar NickSdot
NickSdot - comment - 10 Jan 2020

Wasn't worried about any B/C changes. My question was more regarding how this update process to migrate the old values could look like. Since the quoted one from my previous comments will not work without changing the MySQL config manually (as super) itself. Isn't it?

avatar wilsonge
wilsonge - comment - 10 Jan 2020

Not to the best of my knowledge you can still run these queries mentioned without needing to modify any mysql config

Add a Comment

Login with GitHub to post a comment