We purposefully don't ship core's composer.json and composer.lock files as right now we really do not support someone adding additional libraries to the libraries/vendor directory and altering the core distributed library manifest. For someone that needs them, they can grab those files from the release tag here on the repo, but I would not include these files into the core package or do anything at the moment that promotes altering the Composer directory (especially as this will just be overwritten on an update anyway).

But it works at current time.

Using "composer require" and using libraries with Joomla autoloader

And it causes issues at current time.

As described in the issue there is conflict on updating Core or "composer update".

Yes, overwriting on update is a point. Let's make composer.json.dist so people will always know what to look for?
Forcing users to install composer somewhere else and manually add autoloading is just not right.

Either way I still would not add to the package any file that encourages developers to change the libraries/vendor directory. Our upgrade process will break your local install, there's no way around it. Having Joomla in a state where developers can freely add packages to that directory is a lot more than just including the Composer manifest in the package, that's the main point I want to make here. There are a lot of other things that need to be taken into consideration before we either remove the exclusions for composer.json and composer.lock from the packaging process or include a file like this, including how to handle an upgrade when developers are modifying the local Composer installation and causing the directory's contents to be out-of-sync with the expected core state. Until we figure that out, changes in that directory basically equate to a core hack.

So using composer is a core hack, hm?

Our upgrade process will break your local install

Yes, until I call composer update. And if I include all lines from provided composer.json.dist into composer.json everything will be back up and running.

Well if try to think further: updater can look into composer.json.dist look into composer.json and if second file exists update it by merging both files.
And in the end of all trying to exec('php '.JPATH_LIBRARIES.'composer.phar update) if composer.phar exists.

Still not that simple.

1) The upgrade process simply extracts a ZIP package into the filesystem. It doesn't make checks for things in the filesystem to make decisions on what to (or not to) extract. We'd have to change the upgrade process to extract the ZIP package to a temp directory then move the contents of that temp directory to the production paths to make that work (versus the current process of extracting the ZIP direct to production paths), that's going to add time and resource overhead to the upgrade system. It's feasible to do (I've done it with other projects I refuse to be associated with anymore and wish I could have my name removed from completely), but you do have to be aware of that added time/resource use and adjust the upgrade handling code for it.

2) The exec() function (and other similar functions) are commonly blocked on hosts, so triggering a system command like that isn't an option. So to trigger a composer update in that manner, we'd essentially have to distribute the Composer application and its libraries as part of Joomla.

3) Composer's install/update process commonly takes a fair bit of time (my current Symfony project takes over two minutes to run an update command), to the point that it would time-out on a large majority of hosts. This would leave a site in a broken state.

4) NEVER have an executable PHAR in your web directory. This just smells like trouble.

So using composer is a core hack, hm?

No. Altering the core distributed Composer install (in libraries/vendor) is a core hack. Providing your own Composer managed library in another path isn't. All you're really doing to use that separate library is adding another autoloader to PHP's queue to point to that path.

Look, I want to find a solution that allows developers to add stuff to the main Composer path. We couldn't come up with something when we started down this road almost two years ago and we still don't have one. Until we come up with a safe solution that doesn't cause an upgrade to break sites, I can't in good faith suggest merging anything to core that makes it easier/possible to modify the Composer installation shipped with core. I'm legitimately not trying to block this because I don't think devs should be able to touch the Composer install, but right now Joomla is not in a state where it can support this type of system interaction so we need to stick with the stance of not encouraging or supporting it until we have found the right solution. If you really choose to go against this advice though and make alterations to the Composer installation, instead of having a half-complete (and duplicated) libraries/composer.json.dist you'd be better off downloading the composer.json and composer.lock files directly from the repo here.

Can you take a look at this sketch
There are options to go further, not a concrete approach, just quick sketch, but seems like it's working.

About timeouts or anything like this:
I believe it will install at least some packages. And we can ping script with ajax until response will be 200?

I'd like to make a point one more time:
With this scenario no need in changing installer process.
Just running the script and it will pull all dependencies and reinit autoloader.

Irrespective of @mbabker comments above your composer.json file is incorrect
For example

   "simplepie/simplepie": "^1.4",

will update a joomla install to a version of simplepie that is greater than the one we currently ship with.

Anything that updates a joomla install to a version different to the one than the version the joomla install ships with cant be a good idea. It would be impossible to support end users as we wouldnt know what version of the libraries they are using.

Ok, no problems - we can go with concrete version.
That's not the point. The point is shipping .json.dist file and merging it's content into .json if it's exists. So composer update would not remove joomla needed libraries.
And we also can simplify devs life by doing update (to regenerate autoloader) by joomla if composer file exists.

Other way you Joomla will rewrite autoloader, or composer will uninstall libraries.

So honestly on this whole Composer thing, I would seriously look into how Drupal has dealt with it. https://www.drupal.org/docs/develop/using-composer/using-composer-with-drupal looks like the central resource to start from at an end user perspective, someone will need to spend some more time researching to find their workflows and upgrade processes dealing with the Composer dependencies since as that document states their vendor directory is not checked into version control, meaning if they do package the Composer installation the approaches they'll have taken will be vastly different to our current setup.

The libraries now can be freely updated because our base CMS manifest doesn't use explicit version numbers except for a few specific cases. So what this basically says is that the application is expected to work with certain minimums and feasibly anything newer. As an example the Registry dependency says 1.4.5 is the minimum supported version and we have 1.5.2 in the CMS right now, our manifest says that any newer version should still be supported unless there is a bug or B/C break in future releases of the upstream package on the same major version branch (same goes for all packages; so for example with our manifest as is someone could not pull in Symfony's YAML component on a 3.x release even though from an API perspective in the Registry package it is 100% compatible). If we want to address the "issue" of developers being able to freely upgrade those dependencies, the only option is to manually lock the package versions in the Composer manifest and as part of the update process we change that as well. That's really not efficient (I don't know what best practice is for that actually as it pertains to applications versus libraries).

Anyway, looping back to what this PR is all about. I still suggest it being closed for the time being. Joomla does not support developers altering the Composer installation and upgrades will break it for those who are altering it. An issue should be opened (and stay open) as a collaboration space to work through this, and honestly one should have been open since we merged the Composer structure into the 3.4 development branch something like two years ago.

@alex7r As for your sketch, at a quick glance it does look like it would work, but honestly I feel it would only be efficient for command line based upgrades (which Joomla doesn't support either). Composer's processes aren't optimized to deal with the restrictions of HTTP requests, that's part of the reason I feel even if we could address this issue we couldn't add a hook to the update system to automagically try to upgrade the Composer installation. Maybe one of the third party integrations for Composer helps to deal with this, I can't say I've looked into them though.

Just to give an idea, I went ahead and ran an update of all the development and production dependencies from the current staging branch with Composer's profiling enabled, the full output of that is at https://gist.github.com/mbabker/e1236435c6113058a887844a866a1f77. Now, the results of this are partially skewed because of the profiler, output verbosity, and the fact I have most of these packages in cache already (so they don't need to be downloaded which wouldn't be the case on a lot of servers). As you can see this process is already eating a lot of memory and goes just over 30 seconds, which is the default timeout if I'm not mistaken for a PHP web request. So we already have memory and timing issues to cope with running it as one single command.

@mbabker Well, serious? May be only from Cli? The whole sketch is about doing without Cli, from Http request only.
I start to believe that no attention paid to content of my comments. I believe my English is not SO bad... :sad:

Timeout: timeout can affect you when you redirect user to this page. And if you request it with xhr or any - you can trigger it multiple times until you get 200 code response. It will update 2 of 10 and go timeout on first request, 2+2 of 10 on next and further until 10 of 10. Profit.

One again points of this issue:

• vendors are shipped with Joomla installation.
• .json.dist shipped with installation
• on update: regular flow
• after update ( here and next only if .json exists) .json.dist merged into .json
• test if we can exec or if phar exists:
• - phar - use php script
• - exec - use exec
• if there is no phar available and exec is not permitted: alert user to composer update

Yes, only from CLI. Composer isn't designed to be run on shared hosts from a web request, you're fighting with a lot of crap configurations. So now you have to do a lot of workarounds just to include this into core as a default behavior; things like updating one package at a time. What you're proposing might work on web when you as a developer have more fine tuned control over the server environment, but as a generalized behavior not so much.

I'm not trying to talk past you, disrespect you, or anything like that. But speaking quite frankly, Joomla is not in a technical position to support developers adding their additional libraries to the libraries/vendor directory. Any proposal that does something to make it easier for developers to do so therefore isn't something I can recommend that is accepted into core. That is the underlying issue here. Adding this one file doesn't fix anything. We should not place users in a spot where an upgrade can break their sites because someone is changing the vendor directory (right now this is a core hack because Joomla does not support this behavior and actually discourages it) and a core upgrade overwrites those changes and breaks something else, that just makes Joomla look bad.

One additional point. The inclusion of a composer.json(.dist) alone does nothing. There are other files that are needed to ensure the integrity of the vendor directory and if those aren't correctly tracked and kept in sync it will cause issues, this mainly is the composer.lock file but the libraries/vendor/composer/installed.json file IIRC can be problematic too.

So again, this is not just as simple as adding a file to the package and adding a hook to the upgrade system. There is a lot more that has to be thought through on this for it to work sufficiently enough to be included in core. Until we have addressed all of those issues sufficiently, anything else to me is just asking for trouble.

I'm closing this PR as it will not be merged for sure.
Please feel free to continue the discussion either here, in a new issue or on https://groups.google.com/forum/#!forum/joomla-dev-cms (where it imho makes most sense).

Ok, wait a minute. I believe you're looking quite too straight into this one. Read whole message please before thinking that something statement is wrong...
And I don't know if you will be able to checkmark points, but I will leave them and ask you to check point which you understand and agree with.

Once again:

• I don't propose "customizing or editing" Joomla installation process, but "optionally extending" it.
• Joomla! ships everything like right now (vendors are shipped in zip)
• Joomla ships .json.dist and not .json
• Joomla updates .json from .json.dist on JUpdate
• We don't have to care about pure hosting or security settings because all this will affect only those who already used composer require by them selves, which means those people already done (or at least can) configure their hosting.
• I do not propose to include composer as pivot of installation process and something that will be widely used by extensions 3rd-party developers. (Not it this PR and not now)
• No one except end-developers and site administrators will be able to set up use of composer, which means that only those people can break vendors. And they can do it only if the have intentions to do it.

All the above means:

• We don't force some one into something.
• We help those who forced themselves into using vendors directory with composer.

@Bakual Google - is a great company with great products. But Groups is 👎