? ? Pending

User tests: Successful: Unsuccessful:

avatar richard67
richard67
7 Mar 2021

Pull Request for Issue #32542 (part).

Summary of Changes

This pull request (PR) removes display width modifiers for integer data types in DDL statements in all SQL scripts for MySQL and MariaDB databases.

It is only relevant for MySQL and MariaDB databases, i.e. not PostgreSQL.

It's the J4 equivalent of PR #32606 for staging. See that PR for a detailed description of the reasons for this change.

In addition to the change for the display width modifier, this PR fixes also old null dates being used in SQL scripts for the integration tests. I had to touch the script for MySQL anyway, so I decided to fix that, too, also for PostgreSQL. This change will not need any tests beside the integration tests passing in drone.

Testing Instructions

Requirement

You need a MySQL or MariaDB database.

Testers please report back which of the following 2 kinds of databases you have used for the test:

  • MySQL database server with versions lower than 8.0.17 or any server version of MariaDB
  • MySQL database server with version 8.0.17 or later

If you have both scenarios available, please test both.

Test 1: Reproduce the issue

This test is optional and can be only executed when having a MySQL database server of version 8.0.17 or later.

If you don't have that kind of MySQL database server, skip this test.

During my tests I wasn't able to configure my MySQL 8.0.20 server in a way that the deprecation warnings for the display widths could be shown in the server log file, and you can believe me I've tried a lot of things.

Furthermore, it appeared that when importing an SQL script using display widths for integers in DDL into phpMyAdmin, there are also no warnings shown.

So if you want to see the issue you have to do as follows:

  1. In phpMyadmin, select a database.

  2. Copy the first CREATE TABLE statements for the #__assets table (or any other which contains display widths) from file installation/sql/mysql/base.sql of the 4.0-dev branch (i.e. not from this PR) into the SQL command window.
    Hint: You can use also several statements at once, but you should not use the complete content of the SQL script since in this case phpMyAdmin (or the server?) seem to suppress the warnings due to their high count.

  3. Execute the SQL in the SQL command window (no need to replace the table name prefix before).

Result: You get warnings Warning: #1681 Integer display width is deprecated and will be removed in a future release..

  1. Remove the table created in the previous step.

  2. Repeat steps 2 and 3 but using a statement from the installation/sql/mysql/base.sql file of this PR.

Result: Don't get the warnings from step 3.

Test 2: New installation

  1. Make a new installation with the patch of this PR applied, using an empty database.

  2. Login to backend.

  3. Go to "System-> Manage -> Database" and verify that no database problems are shown.

Result: No database problems found (except of one problem about database version not matching the CMS version when using the installation package created by drone for this PR).

  1. Using a tool like e.g. phpMyAdmin, check data type of columns with integer data types.

Result:

  • On MySQL server versions lower than 8.0.17 or any version of MariaDB server, the display width values of the integer type columns have the default values for that data type. See the summary of changes of PR #32606 for the complete list.
  • On MySQL server versions 8.0.17 or later, the integer type columns don't have any display width values.

Test 3: Update from Joomla 3.10

The starting point is a clean Joomla 3.10 installation (3.10-dev branch or latest 3.10 nightly build).

  1. Login to backend.

  2. Make sure to see any database errors or warnings by switching on "Debug System" and setting "Error Reporting" to "Maximum" or "Development" in Global Configuration.

  3. Update to current 4.0-dev + the patch of this PR applied by using one of the following 2 methods:

  1. While the update runs, verify that no SQL errors or warnings are shown in backend or in your MySQL or MariaDB server's log file or in the PHP error log.

  2. Go to "System-> Manage -> Database" and verify that no database problems are shown.

Result: No database problems found except of one problem about database version not matching the CMS version. This is normal when using updates package created by drone for PR's.

  1. Using a tool like e.g. phpMyAdmin, check data type of columns with integer data types.

Result:

  1. Display widths of database columns which existed already before the update and which haven't been modified for other changes haven't been changed.
  2. New tables have been created with display widths for columns with integer data types depending on the type of database:
  • On MySQL server versions lower than 8.0.17 or any version of MariaDB server, the display width values of the new integer type columns have the default values for that data type with this PR applied, see above in the summary of changes for the complete list.
  • On MySQL server versions 8.0.17 or later, the new integer type columns don't have any display width values with this PR applied.
  1. The same as 2. applies to database columns with integer data types which have been modified for other purpose during the update, e.g. type change or change of default value or null values.

Actual result BEFORE applying this Pull Request

SQL scripts use the unnecessary and for MySQL versions 8.0.17 or later deprecated display width modifier for integer data types.

Expected result AFTER applying this Pull Request

No display width modifier is used for integer data types at any place in SQL statements.

After a new installation on MySQL database servers with versions lower than 8.0.17 or any version of MariaDB servers, database columns with integer data types have the default display widths for the particular data type. On MySQL database servers with versions 8.0.17 or newer, such database columns have no display widths.

When updating from 3.10, display widths haven't been changed for database columns which existed already before the update and haven't been modified by the update for other changes than display widths. For newly created or modified columns, the above statement for new installations applies.

Documentation Changes Required

None.

Other information

@wilsonge When PR's #32606 and #32607 will be merged up from staging or 3.10-dev into 4.0-dev after this PR has been merged, it will cause merge conflicts in the installation/sql/mysql/base.sql file. In this case use the file from the 4.0-dev branch, there should not be any changes on this file coming from staging or 3.10-dev as far as I know.

avatar richard67 richard67 - open - 7 Mar 2021
avatar richard67 richard67 - change - 7 Mar 2021
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 7 Mar 2021
Category SQL Administration com_admin Installation
avatar richard67 richard67 - change - 7 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 7 Mar 2021
avatar brianteeman
brianteeman - comment - 8 Mar 2021

Other than code review and installation checks is there any other way to test? Subject to any side effects from the vaccination I should have time to test this tomorrow

avatar richard67
richard67 - comment - 9 Mar 2021

@brianteeman it’s hard to test because during installation we don’t get these deprecated warnings. So the remaining work here to be done is to find some test description. Maybe you can test my other PR #32605 ? That one is ready.

avatar brianteeman
brianteeman - comment - 9 Mar 2021

Ok will do.

Btw I spotted this originally because the SQL editor I use (heidisql) alerted me to the deprecation.

avatar richard67 richard67 - change - 13 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 13 Mar 2021
avatar richard67 richard67 - change - 19 Mar 2021
Labels Added: ?
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar joomla-cms-bot joomla-cms-bot - change - 20 Mar 2021
Category SQL Administration com_admin Installation SQL Administration com_admin com_banners com_contact com_finder com_newsfeeds Installation
avatar joomla-cms-bot joomla-cms-bot - change - 20 Mar 2021
Category SQL Administration com_admin Installation com_banners com_contact com_finder com_newsfeeds SQL Administration com_admin com_banners com_contact com_finder com_newsfeeds Installation Unit Tests
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
Labels Added: ?
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
Labels Added: ?
Removed: ?
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
Labels Added: ?
Removed: ?
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
Title
[4.0] [WiP] Remove display widths for integer data types from SQL scripts for MySQL and MariaDB databases
[4.0] Remove display widths for integer data types from SQL scripts for MySQL and MariaDB databases
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67 richard67 - change - 20 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 20 Mar 2021
avatar richard67
richard67 - comment - 20 Mar 2021

@brianteeman Ready for test. See also PR's #32606 and #32607 .

avatar brianteeman
brianteeman - comment - 20 Mar 2021

Will try to find some time tomorrow. My time is limited right now

avatar richard67 richard67 - change - 22 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 22 Mar 2021
avatar richard67 richard67 - change - 24 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 24 Mar 2021
avatar richard67 richard67 - change - 24 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 24 Mar 2021
avatar richard67
richard67 - comment - 24 Mar 2021

I've simplified testing instructions. Ready for testing.

avatar richard67 richard67 - change - 24 Mar 2021
Labels Added: ?
Removed: ?
avatar richard67 richard67 - change - 28 Mar 2021
Labels Added: ?
Removed: ?
avatar richard67 richard67 - change - 28 Mar 2021
The description was changed
avatar richard67 richard67 - edited - 28 Mar 2021
avatar richard67
richard67 - comment - 28 Mar 2021

@brianteeman As the other 2 PR's already have been tested, only this one here remains for the display width issue. Maybe you can find some time for testing?

avatar richard67 richard67 - change - 25 Apr 2021
Labels Added: ?
Removed: ?
avatar richard67 richard67 - change - 25 Apr 2021
The description was changed
avatar richard67 richard67 - edited - 25 Apr 2021
avatar richard67 richard67 - change - 25 Apr 2021
The description was changed
avatar richard67 richard67 - edited - 25 Apr 2021
avatar richard67
richard67 - comment - 30 Apr 2021

@brianteeman Do you think you can find the time to test this PR here? I will care for merge conflicts later, and if it is merged before the upmerge of the rest from 3.10-dev, it could even safe @wilsonge time because in the SQL files he can just say "accept yours" for the merge conflicts. I would not ask if it was not a PR for an issue created by you.

avatar brianteeman
brianteeman - comment - 30 Apr 2021

sorry forgot all about this one. looking at it now

avatar brianteeman
brianteeman - comment - 30 Apr 2021

Test1

mysql 8.0.19 success

Test2

clean install with the download from this pr
installed and then installed blog sample data
####result
image

image

(i have not run fix structure)

avatar richard67
richard67 - comment - 30 Apr 2021

@brianteeman That's the usual version not matching database problem after an update with a PR package. I've forgotten to mention that maybe. The result looks ok to me.

avatar brianteeman
brianteeman - comment - 30 Apr 2021

ah - ok thats good to know.

run out of time for today to run test3

avatar richard67 richard67 - change - 30 Apr 2021
The description was changed
avatar richard67 richard67 - edited - 30 Apr 2021
avatar richard67
richard67 - comment - 30 Apr 2021

Thanks so far. I've updated the testing instructions by a hint at the 2 places where it's relevant.

avatar richard67 richard67 - change - 30 Apr 2021
The description was changed
avatar richard67 richard67 - edited - 30 Apr 2021
avatar brianteeman
brianteeman - comment - 1 May 2021

as far as i can tell test3 was also successful

avatar brianteeman brianteeman - test_item - 1 May 2021 - Tested successfully
avatar brianteeman
brianteeman - comment - 1 May 2021

I have tested this item successfully on 634e0c9


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32608.

avatar joomdonation
joomdonation - comment - 2 May 2021

Could we have more detailed testing instructions for test 3:

Display widths of database columns which existed already before the update and which haven't been modified for other changes haven't been changed.

What table and what fields for that table to check, please? Just a sample table should be enough

New tables have been created with display widths for columns with integer data types depending on the type of database:

Again, what table. Just a table to check should be enough.

avatar richard67
richard67 - comment - 2 May 2021

Display widths of database columns which existed already before the update and which haven't been modified for other changes haven't been changed.

What table and what fields for that table to check, please? Just a sample table should be enough

What table: Those which exist on J3 and J4, like most of the core tables, e.g. #__assets, #__categories, #__content, #__extensions, ... any of these which has some columns with integer data types.

What fields: All with an integer data type, which can be tinyint, smallint, mediumint, int and bigint.

As said in the description, check also the description of the meanwhile merged J3 PR #32606 .

It needs to check that these were not changed to the default display widths on your MySQL 5.7 but keep the display widths they had before the update.

Default display widths see the values in brackets:

  • tinyint(4)
  • tinyint(3) unsigned
  • smallint(6)
  • smallint(5) unsigned
  • mediumint(9)
  • mediumint(8) unsigned
  • int(11)
  • int(10) unsigned
  • bigint(20)
  • bigint(20) unsigned

New tables have been created with display widths for columns with integer data types depending on the type of database:

Again, what table. Just a table to check should be enough.

Tables which exist in J4 but not in J3.10. Should be easy to find those. They are easy to find by checking the update SQL scripts for "CREATE TABLE" statements, except of the one script where we modify lots of tables for com_finder, this drops tables and later creates them again. You will find then all tables for workflows, mail templates, webauthn and csp, i.e. for the new functions added in J4. On your MySQL 5.7, all these tables should have the default display widths mentioned above for columns with integer data types.

The easiest way to check all the above is to make a database export into an SQL file of only the structure but not the data, using a tool like phpMyAdmin, one time before and one time after the update, and compare these 2 files with a good code comparison tool like e.g. Beyond Compare. You will see which tables are new in J4, and you will see the differences for the existing tables.

avatar wilsonge
wilsonge - comment - 2 May 2021

I'm merging this so I can do the merge conflicts from staging on the relevent PRs from staging/3.10 - however would be good to see you finish the testing @joomdonation as additional verification that everything has been done here.

avatar wilsonge wilsonge - close - 2 May 2021
avatar wilsonge wilsonge - merge - 2 May 2021
avatar wilsonge wilsonge - change - 2 May 2021
Status Pending Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2021-05-02 22:16:32
Closed_By wilsonge
Labels Added: ?
Removed: ?
avatar richard67
richard67 - comment - 2 May 2021

Thanks everybody.

Add a Comment

Login with GitHub to post a comment