I disagree with this. If extensions dont follow best practice then the
user should edit the file themselves. There are good reasons not to index
those folders.

Using your logic we should make everything 777 just because it would remove
a problem with bad code and/or bad servers

Could you clarify what the good reasons are?

Your second paragraph is erroneous :P

Nick, love your new article but I am a little worried about Google indexing PHP files which could be a security issue. I'm not overly conservative with this kind of stuff, but do advise that we need to carefully consider the potential swift downside to this. Kunena version 4 (currently in the making) has recognized this issue and is swapping it's CSS to the media folder so that it can be indexed. But as mentioned others will probably be slow to make this change. So I guess the hard part is making this more commonly known to developers.

Extensions have to use media folder. If they don't is not Joomla's problem. In fact I'm happy with Google change because it will force developers to do it right.

So big -1 here

I start count #6361, #6702, #6098

I think someone really need to write blogpost about this lazy programmers (who lazy to place css/js/images in the right place)
or just about "what to do if your extension ignored by Google"

Like this one?
https://www.babdev.com/blog/139-use-the-media-folder-allow-overridable-media

On 25 April 2015 at 10:04, Fedir Zinchuk notifications@github.com wrote:

I start count #6361, #6702, #6068

I think someone really need to write blogpost with about this lazy
programmers
or just about "what to do if your extension ignored by Google"

—
Reply to this email directly or view it on GitHub.

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

yes, and + some info about why it is important from google/seo point of view (like in the initial comment here)

I think not everyone understand all reasons why need to place css/js files into /media folder,

https://docs.joomla.org/Development_Best_Practices#Where_should_I_place_JavaScript.2C_CSS.2C_and_Image_files_that_belong_to_my_Component.3F needs some attention to inline the contents of my blog and expand on the recent SEO based emphasis on why this is so important (if it doesn't become a full page in itself; my blog post was really a reaction to seeing too many devs not using JHtml for media).

@JoshuaLewis, thanks man!

PHP files in Joomla, including extensions on the JED, have a defined('_JEXEC') or die statement that prevents the contents of the file being exposed. Even if Google were to find it, the contents would be completely blank and they wouldn't index it.

That being said, if it's desired to not allow the indexing of php files, a derivative of the following could be used:
Disallow: /*.php$

The same can be done for XML.

@phproberto , But what's the point of the rule in the first place? Is it serving its purpose in 2015 when things have changed significantly since its first implementation?

Or are you saying that the reason for adding SEO rules that actually might hurt SEO for users is to force developers to use the media folder? :P

Using the media folder has more benefits than SEO (see the blog post linked above). In our system that gives flexibility to override those files at a template level. So placing your extension media in modules/mod_whatever essentially says to me as a developer that the person coding the extension doesn't care about standard conventions and deliberately makes it difficult for an end user to customize that extension via standard media overrides. It isn't Joomla's job to enable those folks to ignore standards. If a user really needs those folders open to indexing, they can do so on a per-site basis.

Our code allows enough bad practices. Let's not add more please.

Also, from what I can tell, the robots.txt standard (http://www.robotstxt.org/orig.html appears to be authoritative) does not specify extension style disallows; that is a Google specific thing.

I understand the importance of using the media folder and this RFC isn't intended to negate that. However, I also feel that there has to be sufficiently good enough reason to use an SEO rule that actually hurts SEO and I'm still struggling to come up with one. If there isn't a good reason, SEO rules should be used to improve SEO for the benefit of most, standard Joomla users.

As to the media folder, I'm guessing not every resource should be in there. For example, JCH Optimize compresses and gzip's files, then stores them in the plugin directory. IMO, that's a good spot for them and they shouldn't go into the media folder to be overridable, else lets move the cache folder in the media folder too.

Great point Nick about watching out for average Joomla users who try to use decent practices.

I would actually like to see cache moved to the media folder. Caching is becoming more important with websites these days, hence a common occurrence. If compressed (cached) files are out of view from the search engines we end up running into the responsive design being hidden which is a big minus.

On a side note, I already noticed one of my sites ranking up (SEO) due to the new policy. :-)

Not everything in the media folder is automatically overridable. You still have to load the media with the appropriate JHtml helper. Those helpers are essentially proxies to the JDocument methods with the added logic to find overrides; nothing stops extensions like JCH Optimize from dropping that compressed media into the media folder and loading it with the JDocument API to bypass the overridable aspect of things.

We have defined standards and we should be encouraging developers to follow them as appropriate. At the same time, we don't force a robots.txt file on sites (we have a robots.txt.dist which only tries to move itself to robots.txt on a new install IIRC), so end users can modify this file to suit their site's needs. So if accepted, this PR would only matter to new users with the next release of Joomla; existing sites would still need to manually update their file and sadly a lot of users don't understand what a robots.txt file is (look at the forums after 3.4.0's release and see how many "what's this mean?" posts there are).

Great insight, Michael! I'll try and see if I can add that to docs.joomla

Yeah, it's only for new installs.

@JoshuaLewis , great point on the cache folder. I hadn't thought of that, but looks like several of the template frameworks use the cache folder for their compressed files.

Is it standard practice according to Joomla to place cached CSS/JS files into the cached area? If so, perhaps those practices of the cache location could be re-evaluated. The great part about changing the standard (assuming we do it right) is that it doesn't harm current existing template frameworks.

The cache folder has a lot of uses beyond media. The system cache writes HTML files there for example, and many extensions write cache data there that isn't media related. That isn't a folder I think we can just say "go ahead and index all the things" in. So that would need to be considered in the long run.

For example, JCH Optimize compresses and gzip's files, then stores them in the plugin directory.

I do not see a problem if they will move that files to own /media/plg_system_jchoptimize/cache (or something) folder for that purpose

Someone has to code JHtml:: script and stylesheet to produce a versioned edition (if user wants that) of the css and js to bring them in par with addSriptVersion and addStylesheetVersion! Maybe I’ll do it tomorrow, if nobody isn’t interested ????

It already has versioning support - https://github.com/joomla/joomla-cms/blob/staging/libraries/cms/html/html.php#L362

I also feel that there has to be sufficiently good enough reason to use an SEO rule that actually hurts SEO and I'm still struggling to come up with one.

It's not the rule but the extension devs who are hurting SEO.

I cannot believe that being years developing for Joomla you (extension devs) still don't understand why media folder exists and why/when/how should you use it.

There is no good reason to change the core to allow bad practices. Maybe we should directly delete robots.txt.. I may want to load CSS from my library. And why use an images folder? I want users to store images anywhere!

I've seen extensions load media from almost every folder (nothing yet in logs or tmp luckily). So that isn't too farfetched

Based on all the comments I'm going to close this. As indicated extensions should indeed use the media folder and relevent JHtml method.