Some components install their JS, CSS and images in their own /assets directory instead of using the media tags in the XML manifest.
It may sense to optimize the robots.txt to give the ability for GoogleBot to access theses files.
I suggest to modify the robots.txt with theses lines :
Replace the line :
By theses 3 lines :
Allow: /components/*/assets/css/*.css$ Allow: /components/*/assets/js/*.js$ Disallow: /components/
This way, GoogleBot will be able to validate that Joomla sites are more 'Mobile Compatible'.
More infos in my article on Mobile site optimization (sorry, this is a french article so i give you the auto-translate url) :
for me it is the components issue,
I would ask the component developers to move their /assets to /media
Joomla use this long time ago, and I wonder why they still use /assets
|Closed_Date||0000-00-00 00:00:00||⇒||2015-03-07 19:55:29|
Ok, i understand and i agree to close this PR. Devs should used the media tag!
For the 'Allow' directive, you should notice that the most actives bots (google, bing, ...) support this non-standard directive, and that make sense to optimize the "mobile-fiendly" required by Google.
(see 'Allow' directive : http://www.garstud.com/images/stories/blog/joomla/google_compat_mobile4.png )
And I speak in my article about using many 'User-agent' to be sure to give correct directives to correct bots.
No Code Attached Yet