Background

Google recently updated its Webmaster Guidelines: https://support.google.com/webmasters/answer/35769?hl=en#technical_guidelines

"To help Google fully understand your site's contents, allow all of your site's assets, such as CSS and JavaScript files, to be crawled. The Google indexing system renders webpages using the HTML of a page as well as its assets such as images, CSS, and Javascript files."

Google has made clear statements that websites should not be preventing robots from accessing stylesheets and javascript files to assess the quality of pages and their suitability for different devices.

The JUI CSS/JS files are currently stored in /media along with similar files from many extensions, which is currently excluded by our standard robots.txt file which ships with Joomla by default. This is in contravention to these guidelines and causes Joomla sites to flag warnings on the Mobile Friendly test: https://www.google.com/webmasters/tools/mobile-friendly/.

This PR removes /media from the robots.txt file, thereby allowing robots to index the contents of this folder and complying with Webmaster Best Practice Guidelines.

There should also be consideration given to removing /images however given the current use of this directory as the primary upload directory for media, I will submit it in a separate PR so that they can be discussed separately.

Possible implications

There could be content within the media directory which users do not wish to be spidered by search engines.

Test instructions

Before applying patch, check /media is in the robots.txt disallow list. After applying, it should not be there.