[#41151] - [5.0] [GSoC22] Rich Snippets - schema.org implementation
- Fixed in Code Base
- 25 Jul 2023
- Medium
- Build: 5.0-dev
- # 41151
- Diff
- shazmasiddiqui:5.0-dev
User tests: Successful: Unsuccessful:
Introduction
To ensure the delivery of high-quality search results, web crawlers rely on structured data that is used by search engines to generate rich snippets. Currently, Joomla! provides a feature for generating rich snippets, but it uses inline microdata that is challenging to modify from the backend since it is hardcoded into the HTML.
However, a proposed solution is to introduce schema.org and JSON-LD implementation to Joomla!. This would allow structured data to be added and configured via Joomla! backend, simplifying the optimization of website content for search engines and improving the visibility of the website in search results.
By incorporating this update, Joomla! users would benefit from an easier and more flexible way to optimize their website's structured data, leading to a better search experience for their users and potentially increasing their website's traffic and engagement.
Summary of Changes
-
Schemaorg System Plugin: This plugin creates a base form for adding schema. It will generate a default form with an empty list to select a schema type, and then generate a form with relevant attributes for that schema type. This plugin will provide a more user-friendly way of adding structured data to Joomla websites, without requiring users to manually code HTML. The plugin will also use JSON-LD markup for schema implementation, making it easier to read and maintain than traditional HTML markup.
-
Schemaorg Plugins: Each schema type is implemented as a separate plugin. It allows for a more modular and extensible architecture. Each plugin is responsible for generating the schema markup for a specific type of content, which allows for more flexibility and customization options for website owners. It makes it easier for third-party developers to contribute to the development of Joomla's structured data capabilities.
Plugins that we currently have are:
- Blogposting Plugin
- Book Plugin
- Event Plugin
- Organization Plugin
- Person Plugin
- Recipe Plugin
- JobPosting Plugin (Thanks to @tfeuerst)
- Database:
- Adds a
#__schemaorgdatabase table. - Adds
#__extensionsentries forplg_system_schemaorg,plg_schemaorg_blogposting,plg_schemaorg_book,plg_schemaorg_event,plg_schemaorg_organization,plg_schemaorg_person,plg_schemaorg_recipe,plg_schemaorg_jobposting.
Future Ideas
- Implement an automation feature to pre-fill common data automatically in the Schemaorg form.
- Enhance the backend interface to allow users to review and customize pre-filled data as needed.
- Implement a Schema Testing feature within the Schemaorg System Plugin to validate the generated schema.
- Conduct thorough testing, bug fixing, and integration testing.
- Prepare documentation.
Testing Instructions
For testing, a fresh installation/config is important. Steps:
- Go to System Dashboard > Manage module > Plugins
- Search for "Schema.org System Plugin"
- Configure the plugin by adding representation schema
- Go to Content > Articles and click on “New” to add a new article
- Select Schema type
-
Fill in all the details
-
Schema is generated and added to the head tag of the html document in the frontend
Testing Generated Schema using Google's structured data testing tool
Generated Schema
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Organization",
"@id": "http://localhost/gsoc22_schema.org/#/schema/Organization/base",
"name": "Joomla Test",
"url": "http://localhost/gsoc22_schema.org/",
"logo": {
"@type": "ImageObject",
"@id": "http://localhost/gsoc22_schema.org/#/schema/ImageObject/logo",
"url": "images/joomla_black.png",
"contentUrl": "images/joomla_black.png",
"width": 225,
"height": 50
},
"image": {
"@id": "http://localhost/gsoc22_schema.org/#/schema/ImageObject/logo"
}
},
{
"@type": "WebSite",
"@id": "http://localhost/gsoc22_schema.org/#/schema/WebSite/base",
"url": "http://localhost/gsoc22_schema.org/",
"name": "Schemaorg Demo",
"publisher": {
"@id": "http://localhost/gsoc22_schema.org/#/schema/Organization/base"
}
},
{
"@type": "WebPage",
"@id": "http://localhost/gsoc22_schema.org/#/schema/WebPage/base",
"url": "http://localhost/gsoc22_schema.org/index.php/articles/moms-world-famous-banana-bread",
"name": "Mom's World Famous Banana Bread",
"description": null,
"isPartOf": {
"@id": "http://localhost/gsoc22_schema.org/#/schema/WebSite/base"
},
"about": {
"@id": "http://localhost/gsoc22_schema.org/#/schema/Organization/base"
},
"inLanguage": "en-GB",
"breadcrumb": {
"@id": "http://localhost/gsoc22_schema.org/#/schema/BreadcrumbList/17"
}
},
{
"@type": "Recipe",
"image": "images/Moms%20World%20Famous%20Banana%20Bread.jpeg",
"name": "Mom's World Famous Banana Bread",
"author": "John Smith",
"cookTime": "PT1H",
"datePublished": "2023-07-11",
"description": "This classic banana bread recipe comes from my mom -- the walnuts add a nice texture and flavor to the banana bread.",
"prepTime": "PT15M",
"recipeInstructions": "Preheat the oven to 350 degrees. Mix in the ingredients in a bowl. Add the flour last. Pour the mixture into a loaf pan and bake for one hour.",
"recipeYield": "1 loaf",
"recipeIngredient": [
"3 or 4 ripe bananas, smashed",
"1 egg",
"3/4 cup of sugar"
],
"nutrition": {
"@type": "NutritionInformation",
"calories": "240 calories",
"fatContent": "9 grams fat"
},
"@id": "http://localhost/gsoc22_schema.org/#/schema/Recipe/1",
"@isPartOf": {
"@id": "http://localhost/gsoc22_schema.org/#/schema/WebPage/base"
}
}
]
}
Test Results
Documentation changes required
- Update the user documentation to include detailed instructions on utilizing the new Schemaorg Plugins.
- Document the process of adding new schema types in the future and how to integrate them with the existing schema implementation.
Acknowledgements
I would like to express my heartfelt gratitude to my mentors @bembelimen, @ditsuke, @anuragteapot, and @rjharishabh for their continuous guidance and availability throughout the project. I also want to thank @wilsonge for his PR, which served as a valuable reference. Lastly, I am grateful to the Joomla! community for their support and contributions.
joomla-cms-bot
- | Category | ⇒ | Unit Tests SQL Administration com_admin Postgresql com_contact com_content Language & Strings Front End |
| Status | New | ⇒ | Pending |
| Category | Unit Tests SQL Administration com_admin Postgresql com_contact com_content Language & Strings Front End | ⇒ | SQL Administration com_admin Postgresql com_contact com_content Language & Strings Front End |
| Labels |
Added:
?
Language Change
PR-5.0-dev
|
||
| Labels |
Added:
Feature
Documentation Required
Removed: ? |
||
| Title |
|
||||||
@brianteeman Could you check your suggested change here #41151 (comment) ? You have added a comment to the suggested change which should not be part of the suggestion:
Am posting it here again because the reqiew comment is meanwhile hidden by default and has to be expanded for making it visible.
oops - it was missing the backticks - fixed it thanks
I can see a lot of work here but I have a few concerns before I think it is ready to merge.
Implement an automation feature to pre-fill common data automatically in the Schemaorg form.
This shouldnt be hard to do as we already have the data eg author, publish date. Nice to have but maybe not a showstopper
Enhance the backend interface to allow users to review and customize pre-filled data as needed.
This for me is a showstopper. On a large, diverse site you may need to enable all the schema plugins. Now in order to see if you have entered any data in the schema you have to review all the plugin forms individually. At a minimum I should be able to tell if there is any schema data already entered and which form it is in.
Breaking existing sites
Before this PR a site would have some schema data automatically. We can debate if it is any good or not but it exists. This PR removes all that automatic data from the layouts and I now will need to update 4000 articles to put back the data that already existed
Template overrides
If a site has created content overrides that include the existing schema data then they will now get an extra (duplicate/conflicting) set of data from the plugins
Categories
Each of the plugins can be set to include/exclude components but on a content rich site that just means they are all enabled (as mentioned above). If however it supported categories then I could for example set something like
Category: Recipes => Plugin Recipes
category: Jobs=> Plugin Jobposting
etc etc
This really would turn the plugins from something that some people might use if they have the time into something they will find easy to use.
I have tested this item
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41151.
I have tested this item
✅ successfully on b873bf0This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41151.
@bayareajenn
so you're not concerned that all your existing articles will lose their existing gsd ?
I dont really understand this studff but I would expect that https://validator.schema.org/ would not give any errors.
urn:spore:id is not a known valid target type for the breadcrumb property
I have tested this item
✅ successfully on b873bf0
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41151.@bayareajenn so you're not concerned that all your existing articles will lose their existing gsd ?
Apologies, what does gsd stand for?
To illustrate the problems created by this PR below are the results from two sites I just created. They are both joomla 5 with blog sample data.
https://nogsd.brianstest.site - this is the nightly build from last night
https://gsd.briasnstest.site - this is the build from this PR
This setup serves as a good example of whatthe differences will be when you upgrade an existing to site to this version with the plugins - without updating all your content.
I see. Looks like there is more testing I need to do before I can mark this successful. I didn't compare anything previous or existing content or anything like that. Clean build with this pr and just added an article to see output. Apparently not helpful.
I have not tested this item.
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41151.
I have tested this item
✅ successfully on b873bf0
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/41151.@bayareajenn so you're not concerned that all your existing articles will lose their existing gsd ?
Apologies, what does gsd stand for?
@bayareajenn GSD was my shorthand for google structured data aka rich snippets.
Basically this PR removes all the exsiting data from all your content and then you have to add it all manually for every article. So out of the box this PR will probably harm your seo unless you invest the time to update every single piece of content.
that is what the screenshot I just posted show.
Note that we had exactly the same conversation with @wilsonge original pr
My PR left things in. It just added plug-in events for modification. The debate there was when template overrides met up against this and what would win and why if I remember rightly.
I agree with all the points you've raised though.
@shazmasiddiqui For the last 2 review comments about duplicate system plugins in base.sql due to wrong conflict resolution when the PR was rebased to 5.0-dev: See my PR for you shazmasiddiqui#1 . Just use the merge button on GitHub to merge it. Would be nice to get a mention in the thank you section of the description of this PR here since I have also helped with SQL stuff in past.
To illustrate the problems created by this PR below are the results from two sites I just created. They are both joomla 5 with blog sample data. https://nogsd.brianstest.site - this is the nightly build from last night https://gsd.briasnstest.site - this is the build from this PR
This setup serves as a good example of whatthe differences will be when you upgrade an existing to site to this version with the plugins - without updating all your content.
@brianteeman I appreciate the missing data and I don't see it a problem to have them added here (like a fallback mode, I tested yesterday a bit and it looks promising) but out of curiosity, do you have an example where the old (current) Joomla! core rich snippet implementation has a visual effect on the SERPs by outlining something?
I wonder, because the outline in Joomla! is super generic and sets everything to an article without any context. I see that in the testing tool there is a quanity difference, but does Google uses this generic article information somewhere?
Would be nice if you have a real world example, so I can do some tests against it, to find the best solution for the "B/C mode".
you are assuming that the current data is only used for serp
Fair enough, any other live use cases you could provide please?
this changes users content without notice and with the only solution being to manually update all their content. we have no way of knowing all the different ways that someone is using this information. it has never been deprecated or even mentioned in an obscure blog post or article that this would be removed. or that every template that uses an override will need to be updated
As I said, this will be fixed. I was just curious about use cases....
As I said, this will be fixed.
must have missed that commitment
Thanks for this nice feature. I have some small annotations.
- Most important, can we please change this to not prebuilt the schema data and save it to a template, we can do this on the fly using a callback cache. Maybe by always activating this cache or introducing a new level for essential changes.
- automatically use data we already have, that might be a bit tricky but adding a new "simple/legacy" plugin which more or less does what we have now would help us for migration
pretty sure there is more but that's what should be done as minimum.
@shazmasiddiqui For the last 2 review comments about duplicate system plugins in base.sql due to wrong conflict resolution when the PR was rebased to 5.0-dev: See my PR for you shazmasiddiqui#1 . Just use the merge button on GitHub to merge it. Would be nice to get a mention in the thank you section of the description of this PR here since I have also helped with SQL stuff in past.
Thanks @richard67 for this PR and for helping me during the project. I am sorry I missed your name in the acknowledgement.
Thanks @brianteeman for your detailed feedback. I am looking into your suggestions one by one.
@bembelimen @shazmasiddiqui It needs to rename the update SQL scripts "5.0.0-2022-12-10.sql" to something newer than "5.0.0-2023-07-12.sql" (= newest update SQL right now in the 5.0-dev branch), e.g. to "5.0.0-2023-07-25.sql"
Thanks @brianteeman and @richard67
@bembelimen Now we have PHPCS errors in drone: https://ci.joomla.org/joomla/joomla-cms/67834/1/7
Sorry about the extra blank lines - I guess they come from a previous suggestion of mine
I'm merging this for alpha3 now and we (@shazmasiddiqui and @bembelimen) will work on the missing stuff later.
Thanks for this awesome work, I think this will bring much better seo support into joomla then anything else we got till now.
Of course there is much todo but to get some feedback it's better to get it in as early as possible.
| Status | Pending | ⇒ | Fixed in Code Base |
| Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2023-07-25 12:29:27 |
| Closed_By | ⇒ | HLeithner |
So much for following existing policies and listening to feedbackl or even having just one successful test never mind two. What even was the point in spending all that time developing this and testing it if its going to be brushed as9ide. This will end up like workflows a large peice fo code added to core with great potential that is left abandoned in an almost unusable state (as per its intended features)
Please add these p-lugins to the Array of core extensions in libraries\src\Extension\ExtensionHelper.php