BEFORE this is merged the suitabole documentation must be available - unlike with the other recent change

hm, need to think, it a huge rush,
just a first thought, maybe can it be a subforlder? eg media/source/blabla

maybe can it be a subforlder? eg media/source/blabla

Sure, that's why i opened this as RFC

BEFORE this is merged the suitabole documentation must be available

There won't be a single PR for this simply because it cannot. There are way too many things happening here:

• majority of scripts converted to ES6, these need to be tested, eg few dozen PRs
• SPA is very loosely supported, we need a solid approach here
• as far as core concerned, there isn't much to document, all the previous npm commands should behave as before. The only difference is that the sources are moved out of the distribution folder, common best practice. The part that needs documentation is how to use these tools for your own code (module/plugin/template/component), but IMO is not a show blocker.

1000 files is a bit hard to review what exactly you did here. Would be easier if it was done in smaller steps. Especially as some changes seem to be unrelated to each other.
For example in one step just move the source files to /media-source (or build/media or whatever - I don't care much). Another step could be to move the template files to media. A third PR could be to add those subgroups to media. The each can be discussed and if tested and if consent is reached it can be merged.

Regardless of that some things I noticed:

• I see you have quite a few index.html files in your PR, I assume to prevent the folder from disappearing. Imho, that is not needed. Just remove the folder. If someone wants to override something he can create those folder if needed.
• I wouldn't changed anything in the generated media folder structure. Why do you want to add component/module/plugin subfolders there? I disagree that it makes navigating easier, in contrary, Imho that is just a disturbing change for all 3rd party extensions which are supposed to follow and site integrators have to adjust as well as the override folders change. Just use media/com_content like we do today.
• Moving template assets to the media folder is a bit twofold imho. We can do that but it comes a bit awkward to me as we alwas said template files are the big boss. If we now start to allow overriding of the files which are supposed to be last stance (and may be overrides itself), this feels wrong to me. For CSS, we already have a possibility to add custom CSS and if you need to change the template speciific JS you're better off with copying the template anyway.

If someone wants to override something he can create those foldesr if needed.

true, and I agree

Just use media/com_content like we do today.

The impact is really minimal here. The reason is that even if you had some layout overrides in your template (J3) those are completely useless in J4, so one way or another if you have to do a new layout override you will start from a core layout which will have already the correct path for the assets

Moving template assets to the media folder is a bit twofold imho

This needs to be balanced. The pro here is that we get the tools to work with any template, and thus the core is providing the right tooling for development. The con is that we change slightly the behaviour of the template css/js (now can be overridable)

The impact is really minimal here. The reason is that even if you had some layout overrides in your template (J3) those are completely useless in J4, so one way or another if you have to do a new layout override you will start from a core layout which will have already the correct path for the assets

Maybe for core that's true, I can't even say. But for 3rd party it certainly will not be true. Especially not when it comes to extensions which are supposed to work both in J3 and J4. There JS and CSS will be the same or very similar. Question still stands what the benefit is. The assets are already grouped by components/modules/plugins due to the prefixed naming convention (com_foo, mod_foo, plg_foo_bar) we use. This looks like a change for the sake of change to me.

This needs to be balanced. The pro here is that we get the tools to work with any template, and thus the core is providing the right tooling for development. The con is that we change slightly the behaviour of the template css/js (now can be overridable)

I don't understand. You expect 3rd parties will use the core tools to build their templates? I honestly doubt that they will. Or you rather mean any (both) of the core templates?
As said, to me it looks wrong if you allow overriding of overrides (template files) ?
It's not a slight change in behavior, it actually would be quite a big one.

Mmmmmh very big and dangerous change... especially for the Joomla! ecosystem of extensions and templates

Introduce media-source which as the name indicates contains the sources

Can we not use the build directory? Why does this warrant its own top level directory (I don't care if it's shipped or not)?

But for 3rd party it certainly will not be true

What are you talking about? Almost all scripts are exposed through an HTMLHelper method. So what will be broken? Also the system directory remains as is. So I'm confused what will be broken?
3rp devs are known that are not following best practices. I'm reviewing people's work and in 2018 I still see <script> tags inside the output of the modules/component. At some point the Extensions Directory must deny publishing such components that break everyone else's code!

I don't understand. You expect 3rd parties will use the core tools to build their templates?

Yes! That is how I understand adding value to the product by providing easy tools to develop on this platform. My point of view, you don't have to agree with me here

to me it looks wrong if you allow overriding of overrides

I think you got this wrong. css folder in the templates still is the ruling folder. I'm not proposing to change this! What is different is that the template's own css file will not be in that folder but rather in the media/templates/...

I don't understand. You expect 3rd parties will use the core tools to build their templates?

Yes! That is how I understand adding value to the product by providing easy tools to develop on this platform. My point of view, you don't have to agree with me here

Do you really expect all the template clubs with their own frameworks to change - I think that's highly unlikely

Almost all scripts are exposed through an HTMLHelper method.

You're thinking about core here. I'm arguing from a 3rd party view. That's where the confusion comes from.
The media folder is not only used by core. It's meant to be used by 3rd party as well.

3rp devs are known that are not following best practices. I'm reviewing people's work and in 2018 I still see <script> tags inside the output of the modules/component.

Those extensions that use the media folder, also use the Joomla API to include the files. Otherwise it makes not much sense to use the media folder. So I'm not talking about 3rd parties who do their own thing. I'm concerned about those who follow the "best practice" proposed by the core example.

Yes! That is how I understand adding value to the product by providing easy tools to develop on this platform. My point of view, you don't have to agree with me here

Stop dreaming ?
They all look at your work and keep their own already existing tooling. Those who don't already have those tools, will also not use the core ones as they don't have the need for it.

I think you got this wrong. css folder in the templates still is the ruling folder. I'm not proposing to change this! What is different is that the template's own css file will not be in that folder but rather in the media/templates/...

Yep, understood that. I didn't get you wrong. But think about it (again from 3rd party). Your change means that it is expected for templates to put their files into the media/template/foo folder. Let's think about it a bit:
What happens if that 3rd party template has overrides already (which is quite common)? The overrides would be in the templates/foo directory and the ones unique to the template (mainly template.css and eventually a template.js) are in the media/templates/foo folder? That would mean its assets are in multiple places, some overrideable and some not. That is of course bad.
So you'd have to add the media/templates/foo folders to the override lookup paths? That's starting to get crazy of course. Imho no option as well.
At the same time you already have the feature of adding a user.css to the (core, and most 3rd party) template which allows you to adjust the template CSS. If you really have to adjust the JS part, then you likely should do a copy of said template anyway. Or you use one of the many plugins which allow you to replace CSS/JS files.
Imho there is absolutely no need for template files to be overrideable.

Do you really expect all the template clubs with their own frameworks to change

Obviously, the big players have their own tools and I don't expect them to follow the core here. But for many developers, small shops this could be a great help or if you prefer a good way to attract devs to use this product instead of something else. Of course for those that are already offering a number of modules/components/plugins/templates, our tools will have 0 effects as they already have a working toolchain that covers their specific needs (distribution, etc). The idea is not to eliminate other peoples tools but provide sanely and simple tools for the newcomers. How do you expect to grow in a world that every half decent project already provides a great cli for RAD?

They all look at your work and keep their own already existing tooling. Those who don't already have those tools, will also not use the core ones as they don't have the need for it.

**I'll skip the first sentence here. **
So according to your perception of welcoming a newcomer is googling out what available tools are out there for coding in Joomla. And again according to your point of view, this is even remotely OK.
Why don't we just shut down this project?
I mean moving forward is something most people don't want. Making the project more approachable to newcomers is also not a consideration. Seriously now: how do you expect this project to survive if you constantly undermine any progress?

Maybe I should stop pushing for any change, as it seems that people are happy with all the well supported, well documented dying technologies that are quite familiar...

So according to your perception of welcoming a newcomer is googling out what available tools are out there for coding in Joomla.

We're talking about the minimising scripts here, right? Those aren't specific to Joomla at all.
I have no clue how the tools would help any "newcomer". Or which tools you're referring to.

Why don't we just shut down this project?

And here the ultimate "shut down" arguments which are completely offtopic start again.

@Bakual please stop commenting, you have no clue what I'm suggesting and all your comments are totally irrelevant...

you have no clue what I'm suggesting

Then by all means explain

What do you want me to explain?
Take a look at the image, carefully:

The media-source holds the uncompiled assets...

I understood that part, and haven't said anything against that part. I'm fine with moving the source files to whatever place you want them.
My comments where regarding the generated files in the /media folder and where the template assets are located. Maybe you should read my comments again?

My comments where regarding the generated files

Those changes affect ONLY components/modules/plugins/templates that will use the tool (eg: mainly the core). Joomla as always is not dictating anything, so you can still have your media/com_sermon/js. Why all this buzz especially if as you and others here find this tool totally useless and you will keep using your own things?
By the way how many joomla tools exist out there for vuejs and how many for compiling ES6?
I guess jQuery still rules...

Those changes affect ONLY components/modules/plugins/templates that will use the tool. Joomla as always is not dictating anything, so you can still have your media/com_sermon/js.

It's not dictating, that's true. But it sets "best practice" by example and "good devs" are expected to follow the conventions. Or do you think it will look nice if there are folders for components/plugins/modules/templates and 3rd parties still put their assets in to media/com_foo?

By the way how many joomla tools exist out there for vuejs and how many for compiling ES6?

Serious question here because I really don't know: Why would I need a "Joomla tool" for compiling ES6 or vuejs? Afaik those are pretty standard jobs to do which are quite unrelated to Joomla.

The idea is going in the right direction, looking for a way to support JavaScript Apps, SPA, etc; and our current set of assorted scripts that Joomla is packing today.

Before accepting a media-source as the right place to locate the source code, I would like to know what are the long-term goals:

• If it is a new location for the current files, I think it is better to avoid the noise of a reorganization. At least now, we know that there are some legacy folders where the uncompressed/min files are stored and other places where extensions store their own JavaScript code.

• If it is a new way to support a new ecosystem of JavaScript Apps, we need to look into the whole picture to define what ecosystem we want to create.

In general, my questions are:

• Are we going to support SPAs (1 extension per page)?
• Do we want a "Gutenberg builder-Blocks extension system"?
• Are we going to support only the current set of random scripts that are defined by the need of the day (and managed with Document->addScript)?

As far as I am aware, the currently supported use case is the legacy method for the regular desktop setup, and support for a modern mobile ecosystem is out of the picture.

Im a fan of the src folder for assets, but I would prefer a resources folder on extension level.
So we keep the assets where they belong.

I would prefer a resources folder on extension level.

That is also another approach but due to the current structure of Joomla this will make the tooling way too inefficient (way too many folders to watch and recurse). But I'm ok if we decide to go that way (by the way since we have backend/front end scatter code which one will be the true source of code?)

Are we going to support SPAs (1 extension per page)?

@anibalsanchez that is the idea and since we already have an app already build on Vue we should expose the how-to so people can follow. Basically, if you clone this and check the media-source/components/media you'll get a feeling what is there and what needs to be abstracted so everyone can start an SPA today (also run npm run build:mediaMan to get the compiled app)

Are we going to support SPAs (1 extension per page)?

hm, I always thought it already like that,
we have components exactly for that purpose ?
(1 component per page)

Yeah, there are always ways to hack your way around.

What a value most of this PR is the idea of trying to introduce a proper method to support them at the core level. I plan to test it as soon as I have time to review it in detail.

Are we going to support SPAs (1 extension per page)?

This PR doesn't magically make core support SPA infrastructure. Likewise, nothing about the core infrastructure today (3.x, not just 4.0) can't support SPA, you just have an ecosystem not written to support it and if you really want to use Joomla in that structure you're stuck doing a lot of heavy lifting on your own as a result.

Do we want a "Gutenberg builder-Blocks extension system"?

Joomla is not designed out-of-the-box as a page builder solution and taking an existing CMS and making such a major editorial workflow change like that, unless you somehow already have major VC and investor demand for it, is nothing short of controversial and polarizing to the existing ecosystem and would cause more harm than good.

As far as I am aware, the currently supported use case is the legacy method for the regular desktop setup, and support for a modern mobile ecosystem is out of the picture.

If you expect to just take anything off the Joomla ecosystem and use it, you're probably right. You're trying to build for a use case that core can support but the manner in which most extensions are packaged and delivered doesn't. There are only so many changes core can make to suit your workflow, good luck convincing the rest of the ecosystem to follow your desires.

I'm slightly unsure about the

template assets moved also to media. The classic way to override assets (eg css, js) still remains as is but with this structure someone can override the default template.css (impossible till now without touching the index.php)

I mean we don't need to override css assets - it's cascading and we support having the custom.css file still at the template level - so it's really just about js assets I guess - and I'm not sure whether there's a major advantage to that or not

Otherwise everything else seems totally sensible to me

I mean we don't need to override css assets

I think my explanation here caused a lot of misunderstanding. Let me explain a little bit the situation
Right now templates use something like:

HTMLHelper::_('stylesheet', 'template' . ($this->direction === 'rtl' ? '-rtl' : '') . '.css', ['version' => 'auto', 'relative' => true]);

This means that the asset (css/js) can be overrided but the override is the source

If we move the files to /media/whatever and use the same command then the source and the override path will not be the same and thus the inconsistency. But we don't have to make this overridable if we don't really want to, so changing the command to:

HTMLHelper::_('stylesheet', 'media/templates/administrator/' . $this->template . '/css/template' . ($this->direction === 'rtl' ? '-rtl' : '') . '.css', ['version' => 'auto', 'relative' => false]);

Will put us to the exact same spot we were before moving the assets to the media folder (template css and js are't overridable).

I guess this clears up what I meant by assets becoming overridable and that is down to us if we want to allow such change, so moving the assets will not change the previous behaviour if we don't want to change it!

I guess this clears up what I meant by assets becoming overridable and that is down to us if we want to allow such change, so moving the assets will not change the previous behaviour if we don't want to change it!

I guess it's more that I can see advantages of moving media to the media directory if the aim was to place all media in a common directory (kinda moving towards a structure of having entry folders like www/ in modern php apps - but ultimately all our overrides would still be placed in css/js - so I'm just struggling to see the benefit of this change.

Afaik we only use the HtmlHelper for the template CSS and JS files so we can make use of the versioning feature and it automatically looks in the correct template folder (you don't need $this->template, which makes copying the template simpler).
It was never meant to support overriding.

kinda moving towards a structure of having entry folders like www/ in modern php apps

Exactly what I had in my mind. Of course this needs the relevant change in the HTMLHelper::include method (one check if /media/templates/$clientId/$templateName exists we use the media for the overrides, if not we use the $clientId/templates/$templateName so 100% B/C).
But I can understand that this is quite a change, so I'm ok if it gets rejected. The only thing that we need to fix really fast is the separation of source/destination and add media folder to .gitignore because right now all contributors are getting a lot of files marked as changed after running npm i.

Let's get the sources separated as this is quite urgent and then we can talk about the restructuring and other exotic ideas. I'm closing this, small steps...