? ? ? Success

User tests: Successful: Unsuccessful:

avatar nikosdion
nikosdion
17 Dec 2014

Executive summary

This PR introduces a command-line interface (CLI) extensions installer. This allows automation of extension installation (new installs and updates) using command line tools. It can be used on various scenarios including, but not limited to, virtual machine provisioning (see Vagrant), automated tests for extension integration, mass updates of hundreds of sites in a hosting environment and so on. The script in this PR offers an equivalent to all of the installation methods allowed by the Extensions Manager in the back-end of your Joomla! site.

Testing instructions

For testing purposes we are going to install the same dummy extension in several ways. After each installation please check that the extension is installed on your site. Then remove it and test the next method. There are eight tests in total, including the short and long command versions. Important: Being familiar with the command line environment is necessary, given the nature of this PR.

Using an XML update stream (equivalent to Install from Web)

Run this script as
php /path/to/site/cli/install-extension.php -w "http://akeebatest.s3.amazonaws.com/dummy/update.xml"
Also try the long command version
php /path/to/site/cli/install-extension.php --web="http://akeebatest.s3.amazonaws.com/dummy/update.xml"

Using an extension package URL (equivalent to Install from URL)

Run this script as
php /path/to/site/cli/install-extension.php -u "http://akeebatest.s3.amazonaws.com/dummy/com_dummy-1.2.0.a1.zip"
Also try the long command version
php /path/to/site/cli/install-extension.php --url="http://akeebatest.s3.amazonaws.com/dummy/com_dummy-1.2.0.a1.zip"

Using a local extension package (equivalent to Install from Package File)

Download the installation package into a directory, let's call it /foo/bar. DO NOT extract it! Now run
php /path/to/site/cli/install-extension.php -p "/foo/bar/com_dummy-1.2.0.a1.zip"
Also try the long command version
php /path/to/site/cli/install-extension.php --package="/foo/bar/com_dummy-1.2.0.a1.zip"

Using a local directory (equivalent to Install from Directory)

Download the installation package AND extract it into a directory, let's call it /foo/bar. Now run
php /path/to/site/cli/install-extension.php -f "/foo/bar/com_dummy-1.2.0.a1.zip"
Also try the long command version
php /path/to/site/cli/install-extension.php --folder="/foo/bar/com_dummy-1.2.0.a1.zip"

Backwards compatibility

No b/c issues are present. This is a new CLI script which doesn't interfere with existing functionality.

Translation impact

One new translation file added (language/en-GB/en-GB.cli_install_extension.ini) with 22 language strings. Expected translation impact: about 8 minutes.

Credits and acknowledgments

This PR is inspired by the CLI extension installation scripts written by Nicola Galgano and yours truly. It seems that many of us are reinventing the same small but very convenient wheel.

avatar nikosdion nikosdion - open - 17 Dec 2014
avatar jissues-bot jissues-bot - change - 17 Dec 2014
Labels Added: ?
avatar chrisdavenport
chrisdavenport - comment - 17 Dec 2014

Looks good (haven't tested it yet), but I think translation strings will need to be added.

avatar waader
waader - comment - 17 Dec 2014

I am testing with current staging branch, #5429 applied, php 5.4.16 (linux/windows), mysql/postgresql:

With linux/postgresql I get the following messages when executing
php install-extension.php -f "/test/com_dummmy-1.2.0.a1.zip":

Installing with method 'folder'

Extracting package file /test/com_dummy-1.2.0.a1.zip
Installing from /test/install_5491bcdcbe19f

Warning: get_html_translation_table() expects at most 2 parameters, 3 given in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 698

Warning: Invalid argument supplied for foreach() in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 700

Warning: strtr(): The second argument is not an array in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 706

Warning: get_html_translation_table() expects at most 2 parameters, 3 given in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 698

Warning: Invalid argument supplied for foreach() in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 700

Warning: strtr(): The second argument is not an array in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 706

Warning: get_html_translation_table() expects at most 2 parameters, 3 given in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 698

Warning: Invalid argument supplied for foreach() in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 700

Warning: strtr(): The second argument is not an array in /var/www/html/joomla33/libraries/joomla/filter/input.php on line 706
Extension successfully installed

The component is installed but gives a "404 component not found" when invoked.

Under windows it seems that the an absolute path to the package/folder is not supported, or am I doing something wrong when I execute
php install-extension.php -p "c:\test\common_dummy-1.2.0.a1.zip" ?

avatar nikosdion
nikosdion - comment - 17 Dec 2014

The dummy component isn't really designed to install on PostgreSQL. To be honest, apart from Akeeba Backup I am not aware of any other major component which can do as much as install on PostgreSQL. And since the PostgreSQL integration was FUBAR in all previous versions we couldn't even properly test our components against it. That said, I do believe that if you try using Akeeba Backup Core instead of the dummy component it will work. If not, we'll need to take a look at why JInstaller breaks under Postgres which is outside the scope of this PR.

Regarding Windows paths, I suspect the problem is the backslash. Try substituting backslashes with forward slashes, i.e. "C:\Somewhere\Foo.zip" becomes "C:/Somewhere/Foo.zip". If this doesn't work let me know. I will have access to my Windows machine tomorrow morning.

avatar nikosdion
nikosdion - comment - 17 Dec 2014

Chris: It seems to me that the other CLI scripts are not translated. If we need to provide translation strings someone will have to tell me where to put the language files. Back-end? Front-end? What will be the naming convention? Something like en-GB.cli_extension_install.ini? And how exactly does Joomla! know which language to use in the CLI when it does not detect the system language? Whatever is the default language of the site? Doesn't this create a whole lot of issues when an English-speaking-only engineer is installing extensions to a Spanish site?

It seems to me that the PLT needs to have a long discussion with the Translation team and decide what's best. My vote would be a new language directory under CLI and a new translation package for CLI apps. Also we should standardise on using –say– the back-end translations for the library and core components in the CLI apps. Basically, we need a translation strategy for the CLI apps because there's currently none that I know of.=

avatar mbabker
mbabker - comment - 17 Dec 2014

FWIW the Smart Search CLI (https://github.com/joomla/joomla-cms/blob/staging/cli/finder_indexer.php) is translated. en-GB.finder_cli in the site language folder.

avatar mbabker
mbabker - comment - 17 Dec 2014

@waader - If you're looking for an extension that will install (and theoretically work) on PostgreSQL or SQL Server, my own should work. Try https://github.com/BabDev/Podcast-Manager/releases/download/2.1.0/pkg_podcastmanager_2.1.0.zip

avatar chrisdavenport
chrisdavenport - comment - 17 Dec 2014

@nikosdion Your code already loads the administrator "lib_joomla" and "com_installer" language files, although I haven't checked to see if it really does load them or not.

But you raise some good questions. Looking through the existing CLI scripts, two do not appear to be translated at all (garbagecron.php and update_cron.php, although to be fair garbagecron.php doesn't appear to have any strings to be translated anyway). There are two that are translated: deletefiles.php loads the files_joomla.sys file from front-end and as Michael pointed out, finder_indexer.php loads finder_cli also from the front-end.

I'd prefer to keep all the language files in one or other of the two current locations rather than creating a new one just for CLI apps, but I'd be open to discussion on that point. The current CLI apps have set a precedent of using the front-end language folder, so I'd go with that for now. A naming convention would be useful and using "cli_" as a prefix makes sense, so I'd suggest cli_install_extension in this case (since the program is install-extension.php).

What would be nice is if we could agree on using --language as a command-line argument for all our CLI apps so that a different UI language can be loaded for the app. That would cover your English-speaking engineer installing on a Spanish site.

What do you think?

avatar nikosdion
nikosdion - comment - 17 Dec 2014

@mbabker and @chrisdavenport: Thank you for the feedback! Here is the CLI script with proper translation.

@chrisdavenport Regarding the --language parameter, I'd rather JApplicationCli detecting the current system locale instead. It's the POSIX way to communicate to CLI applications your language preference.

avatar chrisdavenport
chrisdavenport - comment - 17 Dec 2014

@nikosdion Okay, sounds fair. Thanks for working on this.

avatar nikosdion nikosdion - change - 17 Dec 2014
The description was changed
avatar brianteeman brianteeman - change - 17 Dec 2014
Category CLI
avatar alikon
alikon - comment - 18 Dec 2014

@nikosdion thanks for unify all the small "wheels" in a modern tire.

just one small consideration sometimes i've found usefull to group on a file all the extensions to be installed
so is possible to install a bounch of extensions with one command only

avatar nikosdion
nikosdion - comment - 18 Dec 2014

@alikon I saw that in your code and there's a reason why I did not implement it in PHP. You can install multiple extensions listed in a file without reinventing the wheel. Let's say you have a file with extension download URLs in /file/with/urls. You just have to use this simple command line:

for e in `cat /file/with/urls`; do php /path/to/site/cli/install-extension.php -u $e; done

If you're on Windows and are about to complain that you don't have Bash to play with I'll give you two alternatives: a. Install cygwin or b. Install Git for Windows. They both install Bash. Make your life easier by also installing ConEmu to manage virtual consoles.

The beauty of this solution is that you can spin up a bash console and have access to both bash internal –also many GNU– commands and Windows executables. That's how I maintain my sanity when I have to work with Windows.

avatar alikon
alikon - comment - 18 Dec 2014

i can agree with you, but sometimes you simply are not allowed to install other "software" due to some restrictive policies on the target machine environment (ie. intranet)

avatar nikosdion
nikosdion - comment - 18 Dec 2014

Challenge accepted. The same thing using the plain old Windows command prompt available since the olden MS-DOS days:
FOR /F "tokens=*" %e IN (c:\your\file.txt) DO php C:\PATH\TO\SITE\cli\install-extension.php -u "%e"
No third party software or PHP code required, just mastering the use of the Force... er... command line.

avatar nikosdion
nikosdion - comment - 18 Dec 2014

I forgot to say that FOR /F has many more options. Take a look at http://ss64.com/nt/for_f.html It's easy to have a CSV file with lines like this:

;method,source
p,c:\somewhere\package.zip
u,http://www.example.com/package.zip
w,http://www.example.net/foobar_updates.xml

and write a single for loop to install the whole lot, e.g.

FOR /F "tokens=1,2 delims=, eol=;" %i IN (c:\your\file.txt) DO php C:\PATH\TO\SITE\cli\install-extension.php -%i "%j"

avatar infograf768
infograf768 - comment - 18 Dec 2014

As I was asked by Chris my thought about this, here they are:

As Michael said, we already use: xx-XX.finder_cli.ini
Therefore I suggest
OR to use en-GB.install_extension_cli.ini
OR change en-GB.finder_cli.ini to en-GB.cli_finder.ini (+ related code + delete the older file in script.php) and keep en-GB.cli_install_extension.ini

NOTE:
Any new ini file added in core has incidences on com_localise component. It requires a bit more in the case of files which are not extension-dependent. It is not an issue as we can update com_localise.

avatar nikosdion
nikosdion - comment - 18 Dec 2014

I prefer having "cli" as a prefix instead of as a suffix. It makes it easier for humans to locate files. As for the Finder CLI translation, if you rename it for consistency you'll end up having to do a lot of unnecessary work including letting all TTs know they have to rename the file; write a new Joomla! upgrade script method to remove the obsolete file; create an installation script for each and every language to remove the obsolete file; change the finder CLI script; change the localisation component. I'm not convinced that this is a good way to expend scant resources.

avatar infograf768
infograf768 - comment - 18 Dec 2014

com_localise will anyway have to be updated. Pity nobody thought a bit more ahead when creating finder_cli.ini. I guess that's life.

avatar nikosdion
nikosdion - comment - 18 Dec 2014

If only I had a penny for any architecture decision I've regretted two years down the line...

avatar waader
waader - comment - 18 Dec 2014

@nikosdion I struggled with this line

php install-extension.php -p "c:\test\common_dummy-1.2.0.a1.zip"

under windows. I found out that when applying the filter "path" the value could not be retrieved. So I changed it - for testing purpose - to "raw".

I also change in line 254 "file" to "package". After that changes package installation works under windows.

In line 254 there is a type - see "langauge". And I think that the comma in line 128 is not necessary.

6a496e7 18 Dec 2014 avatar nikosdion Typos
avatar nikosdion
nikosdion - comment - 18 Dec 2014

@waader Thank you, it's fixed now.

Regarding the comma, it is valid PHP syntax. I put the comma in the last item of an array on purpose. It allows us to refactor arrays (change the order of items, add/remove items) without thinking twice about it.

avatar alikon alikon - test_item - 28 Dec 2014 - Tested successfully
avatar alikon
alikon - comment - 28 Dec 2014

@test success

should go with this one #2764 too
so you can setup all from CLI


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/5460.

avatar waader
waader - comment - 5 Jan 2015

Testing the installation of com_dummy-1.2.0.a1.zip and other components using the different methods went fine.

But I got one problem with the installation of acymailing (com_acymailing_starter_v4.8.1_2015-01-05_13-19-06.zip). Doing this

php install-extension.php -f "c:\com_acymailing_starter_v4.8.1_2015-01-05_13-19-06.zip"

gives the following fatal error: Call to undefined method JoomlaExtensionInstallerCli::isAdmin() in .... joomla34\administrator\components\com_acymailing\helper\helper.php on line 863.

On the other hand: the installation is successful in the backend.

Can somebody more knowledgeable assess this error?

avatar wilsonge
wilsonge - comment - 5 Jan 2015

It means they expect an instance of JApplicationCms when you call JFactory::getApplication() which contains that function (https://github.com/joomla/joomla-cms/blob/staging/libraries/cms/application/cms.php#L663). What we actually have is the cli application instance (https://github.com/joomla/joomla-cms/pull/5460/files#diff-f85f920df9966deeeae69694208c8acdR441) so it fatals. Nic this one is one for you to work out a solution for I guess.

avatar nikosdion
nikosdion - comment - 5 Jan 2015

Technically, this is an architectural issue. I can work around it by adding a dummy method in the CLI script’s class. Is that an acceptable solution?=

avatar wilsonge
wilsonge - comment - 5 Jan 2015

How far would this run though. I'm thinking logically the 3 methods that people would likely use in an install script are isSite, isAdmin and enqeueMessage. I mean I know technically it could be anything. But I'd imagine these 3 would regularly appear

avatar waader
waader - comment - 5 Jan 2015

Meanwhile I tested with Kunena 3.0.6 which gives me another fatal error:
Call to undefined method JoomlaExtensionInstallerCli:getRouter() in ... \joomla34\libraries\joomla\application\route.php on line 47

avatar nikosdion
nikosdion - comment - 5 Jan 2015

I guess we have to deal with this issue architecturally instead of going through a patchwork of workarounds. Having an application interface which really includes all methods developers expect an application object to have and refactoring JApplicationCli / JApplicationWeb is the only way to realistically go forward :(=

avatar mbabker
mbabker - comment - 5 Jan 2015

I was just typing something similar; an interface will definitely help here.

avatar nikosdion
nikosdion - comment - 5 Jan 2015

I guess that this would have to wait for Joomla! 3.5 (I can see a probability for b/c issues there) and, as a result, the CLI extensions installer would also need to wait until then.=

avatar infograf768
infograf768 - comment - 6 Jan 2015

@nikosdion
Shall we consider this PR as postponed to 3.5?
I need to be sure as I would have to upgrade com_localise for this new ini.

avatar brianteeman
brianteeman - comment - 6 Jan 2015

It would be a real shame if it was delayed

avatar nikosdion
nikosdion - comment - 6 Jan 2015

Yes, it has to be delayed. There’s no point shipping broken code just to make it through the 3.4 deadline. We need to deal with the JApplication* classes’ architecture before we can include the CLI extensions installer script.

Of course, if you want to use it on your sites you still can: just check out the file and put it in your site’s cli directory. The script is self-contained. I just can’t vouch it will work with any extension as you can read in this issue thread.=

avatar brianteeman
brianteeman - comment - 6 Jan 2015

:(

On 6 January 2015 at 10:08, Nicholas K. Dionysopoulos <
notifications@github.com> wrote:

Yes, it has to be delayed. There’s no point shipping broken code just to
make it through the 3.4 deadline. We need to deal with the JApplication*
classes’ architecture before we can include the CLI extensions installer
script.

Of course, if you want to use it on your sites you still can: just check
out the file and put it in your site’s cli directory. The script is
self-contained. I just can’t vouch it will work with any extension as you
can read in this issue thread.=


Reply to this email directly or view it on GitHub
#5460 (comment).

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

avatar brianteeman brianteeman - change - 27 Jan 2015
Milestone Added:
avatar nikosdion
nikosdion - comment - 22 Jun 2015

Closing this PR as it requires a major core refactor which can't happen before Joomla! 4.0.

avatar nikosdion nikosdion - change - 22 Jun 2015
Status Pending Closed
Closed_Date 0000-00-00 00:00:00 2015-06-22 13:21:19
Closed_By nikosdion
avatar nikosdion nikosdion - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2015
avatar wilsonge wilsonge - close - 22 Jun 2015
avatar wilsonge wilsonge - close - 22 Jun 2015
avatar wilsonge wilsonge - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - close - 22 Jun 2015
avatar nikosdion nikosdion - close - 22 Jun 2015
avatar joomla-cms-bot joomla-cms-bot - change - 22 Jun 2015
Labels Added: ?
avatar wilsonge
wilsonge - comment - 22 Jun 2015

@marcodings You should review for Joomla 4

avatar wilsonge wilsonge - change - 22 Jun 2015
Labels Added: ?
Removed: ? ?
avatar joomla-cms-bot joomla-cms-bot - change - 22 Jun 2015
Labels Added: ? ?
avatar wilsonge
wilsonge - comment - 22 Jun 2015

I've added the label Re-evaluate for Joomla 4 as well

avatar zero-24 zero-24 - change - 24 Aug 2015
Milestone Added:
avatar zero-24 zero-24 - change - 24 Aug 2015
Milestone Added:

Add a Comment

Login with GitHub to post a comment