? ? Pending

User tests: Successful: Unsuccessful:

avatar twister65
twister65
26 Apr 2019

Rebase Pull Request #23320.
This PR aims to back up and restore the Joomla database using export and import commands (joomla / console).

Summary of Changes

See PR #23320, PR #14272 (and PR #10991).

Testing Instructions

Before:

Then:

  • Go to the cli folder of your website.
  • Export all tables and data to the folder :
    php joomla.php database:export --all --folder <folder_path>
  • Import all from folder:
    php joomla.php database:import --all --folder <folder_path>

You can also :

  • Export all tables and data in zip file :
    php joomla.php database:export --all --zip
  • Export a table :
    php joomla.php database:export --table <table_name>
  • Export a table as a .zip file :
    php joomla.php database:export --table <table_name> --zip
  • Import a table :
    php joomla.php database:import --table <table_name>

If you need help:
php joomla.php database:export --help
php joomla.php database:import --help

Expected result

Export/import tables with structure and data.

Actual result

Does not export/import views, stored procedures and triggers (optional for the Joomla database).

Documentation Changes Required

avatar twister65 twister65 - open - 26 Apr 2019
avatar twister65 twister65 - change - 26 Apr 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 26 Apr 2019
Category CLI
avatar twister65 twister65 - change - 26 Apr 2019
The description was changed
avatar twister65 twister65 - edited - 26 Apr 2019
avatar mbabker
mbabker - comment - 26 Apr 2019

These CLI files really should make use of the joomla/console package, and there's really no reason they couldn't be included in the Framework package instead of only in the CMS (though it might take a couple iterations to switch it from single file deprecated CLI things to proper commands but it'd be worth it in the end).

avatar twister65 twister65 - change - 27 Apr 2019
The description was changed
avatar twister65 twister65 - edited - 27 Apr 2019
avatar twister65 twister65 - change - 27 Apr 2019
Title
database importer exporter (rebase)
[4.0] database importer exporter (rebase)
avatar twister65 twister65 - edited - 27 Apr 2019
avatar twister65
twister65 - comment - 28 Apr 2019

Is there a documentation to implement joomla/console ?
The only one I found is:
https://github.com/joomla-framework/console/blob/master/docs/overview.md

If I execute:
https://gist.github.com/twister65/c52a2e5816fd9a0b923b9cd188719d5f
I have:

PHP Fatal error: Uncaught Error: Class 'Joomla\Console\Application' not found in /home/olivier/src/database/Console/ExportApplication.php:11

avatar mbabker
mbabker - comment - 28 Apr 2019

Do you have a code branch somewhere? You should only need command classes,
not an application class.

Check the Framework’s Event, Keychain, Router, and Session packages for
example commands. The CMS repo has some commands as well. And check the
framework.joomla.org repo for a full application example.

On Sun, Apr 28, 2019 at 10:25 AM Olivier notifications@github.com wrote:

Is there a documentation to implement joomla/console ?
The only one I found is:
https://github.com/joomla-framework/console/blob/master/docs/overview.md

If I execute:
https://gist.github.com/twister65/c52a2e5816fd9a0b923b9cd188719d5f
I have:

PHP Fatal error: Uncaught Error: Class 'Joomla\Console\Application' not
found in /home/olivier/src/database/Console/ExportApplication.php:11


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#24733 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACZ7IL2SWSNW25EJ65TBNLPSW6W3ANCNFSM4HI2J7SQ
.

--

  • Michael Please pardon any errors, this message was sent from my iPhone.
avatar twister65
twister65 - comment - 1 May 2019

@mbabker , the code branch is:
https://github.com/twister65/database/tree/2.0-exim
I add command classes in this commit:
twister65/database@c888037

I failed to test these commands with the joomla/console/application.

PHP Fatal error: Uncaught Error: Class 'Joomla\Console\Application' not found in /home/olivier/src/database_test/src/ExporterApplication.php:11

See https://gist.github.com/twister65/c52a2e5816fd9a0b923b9cd188719d5f

We must also read the configuration file to configure the database interface.

avatar mbabker
mbabker - comment - 1 May 2019

For the class not found bit, you'll need to add the relevant packages to the require-dev section of the composer.json file, you should also add them to the suggest section. See joomla-framework/event@e4c851e#diff-b5d0ee8c97c7abd7e3fa29b9a27d1780 for the Event package as an example. Make sure you composer update after adding them otherwise the files won't be in the vendor directory (if you don't have a vendor/joomla/console/src/Application.php file then something's not right).

We must also read the configuration file to configure the database interface.

For the Database package, what you've got in there is just fine for now. When integrated into the CMS, the service layer would have to take care of loading in the correct database connection when the command is run. We can worry about that later.

I'll do a more thorough look at your branch later.

avatar twister65 twister65 - change - 3 May 2019
Labels Added: ?
avatar twister65
twister65 - comment - 3 May 2019

@mbabker , I put the console application in the cli directory. Is this the right place ?
I also updated the testing instructions to use console commands.

avatar mbabker
mbabker - comment - 3 May 2019

Is this the right place ?

Not quite. Instead of adding a new application, you'd register the commands with the existing application (part of the point of the console package is to make it so you can build commands usable in any compatible application, whereas the old concept of JApplicationCli basically meant every console command was its own application).

So for the CMS integration, you're really doing 2 things:

  1. Registering services for the commands.

In libraries/src/Service/Provider/Console.php register the two database commands as services:

$container->share(
	ExportCommand::class,
	function (Container $container)
	{
		return new ExportCommand($container->get('db'));
	},
	true
);

$container->share(
	ImportCommand::class,
	function (Container $container)
	{
		return new ImportCommand($container->get('db'));
	},
	true
);
  1. Registering the commands to the core CLI application.

In libraries/src/Service/Provider/Application.php, add the two commands to the mapping array of the WritableContainerLoader service (this allows the command classes to be lazily instantiated through the container only when they're needed). Then when running php cli/joomla.php you should see the two commands in the list of available commands (this won't work fully until the PR to the Framework repo is merged and a composer update is run on this repo with the additions).

avatar joomla-cms-bot joomla-cms-bot - change - 3 May 2019
Category CLI CLI Libraries
avatar mbabker
mbabker - comment - 3 May 2019

You didn't need that last commit. For the CMS integration, the only changes this PR needs is the changes to the files in libraries/src/Service/Provider as that will handle registering the commands from the Framework's Database package/repository to the CMS application. You don't need to add the command classes to the CMS and you don't need separate files in the cli directory.

The only thing this PR is dependent upon once it gets in that state is the pull request in the Framework repository to be merged then the CMS pulls in the upstream changes.

The only reason to add the command classes to the CMS repo directly is if the CMS 100% cannot use the classes in the Framework package, which from everything I've seen that is not the case.

avatar twister65 twister65 - change - 5 May 2019
The description was changed
avatar twister65 twister65 - edited - 5 May 2019
avatar twister65 twister65 - change - 5 May 2019
The description was changed
avatar twister65 twister65 - edited - 5 May 2019
avatar twister65 twister65 - change - 5 May 2019
The description was changed
avatar twister65 twister65 - edited - 5 May 2019
avatar joomla-cms-bot joomla-cms-bot - change - 5 May 2019
Category CLI Libraries Libraries
avatar twister65
twister65 - comment - 5 May 2019

It is ready for testing with Joomla/Console. See the testing instructions.
@mbabker , thanks.

avatar twister65 twister65 - change - 5 May 2019
The description was changed
avatar twister65 twister65 - edited - 5 May 2019
avatar mbabker
mbabker - comment - 5 May 2019

Upstream PR's merged, this repo just needs the corresponding composer update then this is ready to merge.

avatar alikon
alikon - comment - 6 May 2019

to be honest i didn't test all the options .... sorry i'm lazy ...
but for what i've tested (mostly postgressql)..... please simply merge it

avatar alikon
alikon - comment - 6 May 2019

I have tested this item successfully on 8fa6edf


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/24733.

avatar alikon alikon - test_item - 6 May 2019 - Tested successfully
avatar mbabker
mbabker - comment - 6 May 2019

to be honest i didn't test all the options .... sorry i'm lazy ...
but for what i've tested (mostly postgressql)..... please simply merge it

FWIW I added some functional tests for both the export command and import command on a MySQL build, so between the tests for the import/export class chains and the command tests we should be covered on all the functionality now.

Since all this PR is doing is adding the integration for the CMS' console application, this should be considered RTC as any bugs in either the import or export code, or the CLI commands, are going to be upstream bugs and not a blocker for this PR. The branch just needs to have 4.0-dev merged in again to pick up the autoloader fixes that broke the tests.

avatar alikon
alikon - comment - 6 May 2019

your statements are gold to my ears
i don't have the power to merge it but at least i can mark it as rtc

avatar alikon alikon - change - 6 May 2019
Status Pending Ready to Commit
avatar alikon
alikon - comment - 6 May 2019

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/24733.

avatar twister65 twister65 - change - 6 May 2019
The description was changed
avatar twister65 twister65 - edited - 6 May 2019
avatar twister65
twister65 - comment - 7 May 2019

Try to import in a new blank database (without table). The joomla/console script fails, unlike the previous cli script or Joomla/Console Application.
Question: is this a problem in our context ?

avatar twister65 twister65 - change - 7 May 2019
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - change - 7 May 2019
Category Libraries CLI Libraries
avatar mbabker
mbabker - comment - 7 May 2019

Define "fails".

avatar twister65
twister65 - comment - 7 May 2019

Define "fails":

Whoops, looks like something went wrong.

whoops

  • Export all tables and data to the folder :
    php joomla.php database:export --all --folder <folder_path>
  • Delete and create an empty database with the same name.
  • Import all from folder:
    php joomla.php database:import --all --folder <folder_path>
avatar mbabker
mbabker - comment - 7 May 2019

Put Joomla in debug mode and see what happens. IIRC I set up the Symfony debug handler is configured to show the exception details and backtrace when debug's enabled (but the Console application is also supposed to be catching and processing Exceptions within it so I have no idea why that handler is actually being triggered).

avatar twister65
twister65 - comment - 7 May 2019

Same result with debugging enabled. I have also defined the session handler on filesystem (previously defined on database). Without effect ...

avatar mbabker
mbabker - comment - 7 May 2019

Well, without an Exception to debug things aren't going to go too far. You might need to find one of the catch blocks in the Joomla\Console\Application class and dump the Exception there to see what's going on.

Though if I had to blindly take a guess, given your "empty database" statement, it's probably an Exception being thrown from the drop table call in the import command, but that would then mean it's not a ExecutionFailureException being thrown because the command is catching and handling that.

avatar twister65
twister65 - comment - 7 May 2019

With DatabaseConsole, it works fine.
3bec551
It uses the same commands.

avatar mbabker
mbabker - comment - 7 May 2019

All that means is that the command executes correctly in an isolated context. That doesn't identify if there is an integration issue with the Console package and the CMS or some quirk in the import code that you don't get from a standalone version that has no outside influences. That DatabaseConsole file can't be merged because the intent is all CLI things go through cli/joomla.php and the CMS' console application, there should not be the mess of random CLI files to get executed in 4.0.

avatar twister65
twister65 - comment - 7 May 2019

ERROR: relation "vwe2g_viewlevels" does not exist
debug.txt

avatar joomla-cms-bot joomla-cms-bot - change - 7 May 2019
Category Libraries CLI Libraries
avatar mbabker
mbabker - comment - 7 May 2019

That'd be expected then. Unfortunately a chunk of the CMS outside the install app just requires some of the database to be there, I'm not sure you'd be able to make the import command work on a completely empty database (maybe some of the work in #21452 helps with this scenario).

avatar twister65 twister65 - change - 7 May 2019
The description was changed
avatar twister65 twister65 - edited - 7 May 2019
avatar twister65 twister65 - change - 7 May 2019
The description was changed
avatar twister65 twister65 - edited - 7 May 2019
avatar roland-d roland-d - change - 23 May 2019
Status Ready to Commit Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2019-05-23 06:23:49
Closed_By roland-d
avatar roland-d roland-d - close - 23 May 2019
avatar roland-d roland-d - merge - 23 May 2019
avatar roland-d
roland-d - comment - 23 May 2019

Thank you

@twister65 Would you mind creating a page on the https://docs.joomla.org site documenting this feature?

avatar twister65
twister65 - comment - 26 May 2019

Of course, I created an account and confirmed my email. But I can't edit a new page:

Permission error
You do not have permission to create pages, for the following reason below:
Editing of this page is limited to registered doc users in the group: Email Confirmed.

Maybe someone else can do it, click on the link below:
https://docs.joomla.org/CLI_Database_Exporter_Importer
And copy/paste this text:
JDoc_CLI_Database.txt

avatar alikon
alikon - comment - 27 May 2019

@twister65 docs page done can you double check please https://docs.joomla.org/CLI_Database_Exporter_Importer

Add a Comment

Login with GitHub to post a comment