? ? ? ? Pending

User tests: Successful: Unsuccessful:

avatar wilsonge
wilsonge
1 Jan 2019

This is a base for a web services implementation for Joomla 4

What this has got in

Core

  • Content Negotiation (core provides a basic api integration using the https://github.com/tobscure/json-api library)
  • Plugin based routes to allow custom endpoints to be added
  • Authentication on endpoints using users basic auth (through plugins to allow future oAuth integrations)
  • Exceptions added to allow quality error handling

Integrations/Testing

  • Sample com_content integration
  • Basic system tests

What does this not have in

  • oAuth authentication
  • Ability to turn off web services - although there is an auth layer in front of it
  • Some sort of fundamental change to our error handling (the way we set up errors in JTable and JModel mean we'll likely never be able to have nice error handling with the current model system. In many places we're likely going to need to add 500's just because we won't be able to detect the right exceptions. In the future we should consider a new mvc layer based on the new entity layer https://github.com/joomla-framework/entities but this will take longer and be much harder to implement. The fundamental base application would not be expected to change but extension MVC's will have to to have a nice working integration.

Thanks

  • @mbabker for large amounts of consulting and framework contributions especially around the router as well as for the code for api part of the ComponentInstaller Adapter
  • @muhakh and @cokencorn for their work on their GSOC project which formed a large amount of the base application
  • @isacandrei for work bug testing and working on various next gen parts

Before moaning about a lack of feature X

Please read the specification https://joomla-projects.github.io/gsoc18_webservices/ - this has long been agreed on and is by no means perfect. Think about if it's absolutely essential for a day 1 integration or not. What we have in this PR is already significantly more than wordpress has and is probably equal to what's available in the other top level CMS'

Testing

  • When applying the patch either do a fresh install of Joomla or discover install the new plugins
  • In the specification docs above there's a "installing the latest codebase" menu item which contains sample API calls you can make (currently this is just for com_content)
  • Please ensure with all the library changes that nothing has broken in the backend/frontend too
avatar wilsonge wilsonge - open - 1 Jan 2019
avatar wilsonge wilsonge - change - 1 Jan 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 1 Jan 2019
Category Unit Tests Administration com_content Language & Strings External Library Composer Change Libraries
avatar wilsonge wilsonge - change - 1 Jan 2019
The description was changed
avatar wilsonge wilsonge - edited - 1 Jan 2019
avatar wilsonge wilsonge - change - 1 Jan 2019
The description was changed
avatar wilsonge wilsonge - edited - 1 Jan 2019
avatar dgrammatiko
dgrammatiko - comment - 1 Jan 2019

Merge it!

avatar brianteeman
brianteeman - comment - 1 Jan 2019

Sorry to be pedantic but the correct term according to the w3c it is "Web Services" not "webservices"

As this is new to Joomla its best to get it right from the beginning

avatar wilsonge wilsonge - change - 1 Jan 2019
Labels Added: ? ? ? ?
avatar wilsonge wilsonge - change - 1 Jan 2019
Title
[4.0] Webservices
[4.0] Web Services
avatar wilsonge wilsonge - edited - 1 Jan 2019
avatar wilsonge
wilsonge - comment - 1 Jan 2019

@brianteeman I've amended the language strings and the doc blocks where we refer to web services as a noun. I've left the plugin type as it is - because obviously it can't have spaces and i've found from practical experience with editors-xtd that most people struggle with the concept of removing the - from the plugin class names - so it's more practical to keep the group as a single word I think. Happy to be challenged on that if that's the majority tho

avatar wilsonge wilsonge - change - 1 Jan 2019
The description was changed
avatar wilsonge wilsonge - edited - 1 Jan 2019
avatar mbabker
mbabker - comment - 2 Jan 2019

App needs to be registered in Joomla\CMS\Application\ApplicationHelper::getClientInfo()

(Side note, this method should be redone to use the event system instead of hardcoding a list...)

avatar wilsonge wilsonge - change - 5 Jan 2019
The description was changed
avatar wilsonge wilsonge - edited - 5 Jan 2019
avatar joomla-cms-bot joomla-cms-bot - change - 5 Jan 2019
Category Unit Tests Administration com_content Language & Strings External Library Composer Change Libraries Unit Tests Administration com_content Language & Strings External Library Composer Change Installation Libraries
avatar joomla-cms-bot joomla-cms-bot - change - 5 Jan 2019
Category Unit Tests Administration com_content Language & Strings External Library Composer Change Libraries Installation Unit Tests SQL Administration com_admin Postgresql com_content Language & Strings External Library Composer Change Installation Libraries
avatar Hackwar
Hackwar - comment - 7 Jan 2019

I like this, but we would indeed need at least a way to switch it off. Maybe simply adding a switch in configuration.php?

avatar wilsonge
wilsonge - comment - 7 Jan 2019

@Hackwar addressed all your comments. So I deliberately didn't put in the switch yet because I want it in a separate PR. Reason being - there is an argument that inter-component webservices should always be available. The switch should be a limit for the public facing webservices (I honestly can see this both ways). As a result I want to deliberately NOT have that discussion in this PR and leave it for later and instead introduce the base application here and let that discussion be done in a separate follow up PR (which I commit to doing in order to allow that conversation to take place)

avatar manojLondhe
manojLondhe - comment - 17 Jan 2019

I tried testing this

This worked

  • Get articles list
    • GET {{host}}/api/index.php/article

This did not work

  • Get single article

    • GET {{host}}/api/index.php/article/1
    • {"errors":{"code":500,"title":"Internal server error","detail":"Array"}}
  • Create article

    • POST {{host}}/api/index.php/article
    • {"errors":[{"code":500,"title":"Internal server error","detail":"RuntimeException: Table articles not supported. File not found. in /libraries/src/MVC/Controller/ApiController.php:295\nStack trace:\n#0 /libraries/src/MVC/Controller/BaseController.php(735): Joomla\CMS\MVC\Controller\ApiController->add()\n#1 /libraries/src/Dispatcher/ComponentDispatcher.php(146): Joomla\CMS\MVC\Controller\BaseController->execute('add')\n#2 /components/com_content/Dispatcher/Dispatcher.php(57): Joomla\CMS\Dispatcher\ComponentDispatcher->dispatch()\n#3 /libraries/src/Component/ComponentHelper.php(382): Joomla\Component\Content\Site\Dispatcher\Dispatcher->dispatch()\n#4 /libraries/src/Application/ApiApplication.php(331): Joomla\CMS\Component\ComponentHelper::renderComponent('com_content')\n#5 /libraries/src/Application/ApiApplication.php(108): Joomla\CMS\Application\ApiApplication->dispatch()\n#6 /libraries/src/Application/CMSApplication.php(240): Joomla\CMS\Application\ApiApplication->doExecute()\n#7 /api/includes/app.php(54): Joomla\CMS\Application\CMSApplication->execute()\n#8 /api/index.php(35): require_once('/home/manoj/GIT...')\n#9 {main}"}]}
  • Update article

    • POST {{host}}/api/index.php/article/1
    • {"errors":[{"title":"Resource not found","code":404}]}
  • Delete article

    • DELETE {{host}}/api/index.php/article/1
    • {"errors":{"code":500,"title":"Internal server error","detail":"Array"}}

Here is a postman collection I created and used
(import below url in postman, create postman env, set up host, username, password and use this)
https://www.getpostman.com/collections/cf166ea0211f65ff23dc

avatar dgrammatiko
dgrammatiko - comment - 17 Jan 2019

@wilsonge quick question: does web services require the joomla instance to be in the root folder or will it work on a subfolder?

avatar mbabker
mbabker - comment - 17 Jan 2019

The API app should behave the same as the admin app as far as folder structure goes (since the requests are going to /api/index.php and not through the root /index.php).

avatar sanderpotjer
sanderpotjer - comment - 24 Jan 2019

First off all: thank you @wilsonge and all others worked on this Web Services PR, great work!

We have been looking into this PR this afternoon. In the past we have developed a custom REST API solution to serve Joomla articles for our customers. So as a test-case we are now doing this again, but by using this new core feature (and possibly a custom web services plugin).

A couple comments so far:

1. Documentation
You asked us to read https://joomla-projects.github.io/gsoc18_webservices/, which kind of pushed us into the incorrect direction for testing this PR ;-) The example URLs on https://joomla-projects.github.io/gsoc18_webservices/?specification/chapters/urls-and-routing.md are no longer valid. It might be good to update that page, or have some up to date documentation for people that want to test this PR. And mention that the authentication works via the Joomla user accounts.

2. Requests not working
We confirm the "not working" items mentioned by @manojLondhe

3. SEF support
The GET articles list is working, but it seems that as soon as we turn "Use URL Rewriting" on, or completely disable the "Search Engine Friendly URLs" the API will return a "Resource not found" error response.

4. ACL
Currently, the only authentication to access the API is done by checking the password of a user. We understand that additional API authentication plugins can be created, but would it not be good to add an ACL check as well? We do see a reference to an action "core.login.api" in https://github.com/joomla/joomla-cms/pull/23424/files#diff-45ae31282d7eb57ef5d8dc26df256d9fR286 but that is not implemented further in the web services / Joomla permission interface. We do think it would be a good idea to add that "core.login.api" action and check for the permission. In this way you don't grant anyone with an account on the website access to the API by default. There are situations that only a specific "API users" should be able to access the API. The "core.login.api" is also in line with the offline, admin and site login actions. As a bonus: this might be the solution as well to "turn off web services" by simply not allowing this action for the Public user group.

These are some first comments based on our first impressions. Its great to see Web Services are on the horizon of Joomla 4 and will try to test/contribute as much as possible to make this happen for Joomla 4 :)

avatar joomla-cms-bot joomla-cms-bot - change - 24 Jan 2019
Category Unit Tests Administration com_content Language & Strings External Library Composer Change Libraries Installation SQL com_admin Postgresql Unit Tests SQL Administration com_admin Postgresql com_config com_content Language & Strings External Library Composer Change Installation Libraries
avatar wilsonge
wilsonge - comment - 24 Jan 2019

First of all thanks so much for the detailed feedback and testing!

Documentation

Sorry forgot to update that page. This had the 'correct' URLs on https://github.com/joomla-projects/gsoc18_webservices/blob/master/manual/en-US/about-joomla/installing-the-repo.md but either way I've normalised everything (I've now also expanded the API Auth section). Note that since your test I've added in the version prefix as per the original plan. Hopefully this now reflects what's in the product (once github propagates the commits into the docs)

There's a possibility I might swap the Basic Auth out for a token based system before this goes stable. But for now at least it will be basic auth.

Requests not working

Did you guys clear the libraries/autoload_psr4.php file to autoload the components api sections? Because if you didn't that would definitely cause the issue you're seeing. I have also fixed an issue with the edit and post commands. Looks like I still have a bug in the delete one which I'll look into tomorrow.

SEF support

I thought I'd covered this with https://github.com/wilsonge/joomla-cms/blob/webservices-version2/libraries/src/Router/ApiRouter.php#L117-L128 - might need some help to test this if it's not working as I'm mostly working local and haven't set up the whole htaccess stuff.

ACL

Oops section missing from com_config now added. Hopefully that's the permission you were looking for (note by default I'd picked admin users only but again that can all be negotiated)

avatar wilsonge
wilsonge - comment - 26 Jan 2019

Delete button fixed. That should be all the endpoints working again

avatar wilsonge wilsonge - change - 4 Mar 2019
Title
[4.0] Web Services
[4.0] [NO CACHE] Web Services
avatar wilsonge wilsonge - edited - 4 Mar 2019
avatar yvesh yvesh - test_item - 4 Mar 2019 - Tested successfully
avatar yvesh
yvesh - comment - 4 Mar 2019

I have tested this item successfully on bd17e19

Basic API calls work as expected.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/23424.

avatar wilsonge
wilsonge - comment - 4 Mar 2019

@SniperSister @zero-24 can you guys have a look at RIPS please (carefully) and if it's good i'm merging this

avatar wilsonge wilsonge - change - 5 Mar 2019
The description was changed
avatar wilsonge wilsonge - edited - 5 Mar 2019
avatar wilsonge wilsonge - change - 5 Mar 2019
The description was changed
avatar wilsonge wilsonge - edited - 5 Mar 2019
avatar wilsonge
wilsonge - comment - 5 Mar 2019

Thanks @alikon

avatar wilsonge
wilsonge - comment - 5 Mar 2019

@SniperSister @zero-24 looks like still some RIPS issues - although most have been solved elsewhere

avatar zero-24
zero-24 - comment - 5 Mar 2019

Rips has been cancelled just restarted it here: http://ci.joomla.org/joomla/joomla-cms/15490

avatar wilsonge
wilsonge - comment - 5 Mar 2019

@zero-24 Sorry my bad I started playing around with the API tests -fixed again now. http://ci.joomla.org/joomla/joomla-cms/15495/19 contains rips

avatar wilsonge wilsonge - change - 6 Mar 2019
The description was changed
avatar wilsonge wilsonge - edited - 6 Mar 2019
avatar wilsonge wilsonge - change - 6 Mar 2019
The description was changed
avatar wilsonge wilsonge - edited - 6 Mar 2019
avatar Oracions2410
Oracions2410 - comment - 7 Mar 2019

Hello everyone .. I am interested in this project and wish to participate. Can you guide me please?

avatar Oracions2410
Oracions2410 - comment - 7 Mar 2019

I have exceptional skills in PHP. And a good dose of motivation. Help me please !

avatar wilsonge wilsonge - change - 7 Mar 2019
Status Pending Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2019-03-07 18:16:11
Closed_By wilsonge
avatar wilsonge wilsonge - close - 7 Mar 2019
avatar wilsonge wilsonge - merge - 7 Mar 2019
avatar wilsonge
wilsonge - comment - 7 Mar 2019

Merging. Thanks again to all those who gave input, and the gsoc students who worked on this project. It's hugely appreciated!

avatar Oracions2410
Oracions2410 - comment - 7 Mar 2019

Yes. I just discovered the world of open source and I think Gsoc is a good start!

avatar wilsonge
wilsonge - comment - 7 Mar 2019

@Oracions2410 that's awesome. If you drop one of the GSOC admins on that page a email they'll drop you in some of our chat channels where we can discuss details of the project etc :)

avatar Oracions2410
Oracions2410 - comment - 7 Mar 2019

@wilsonge
Excuse me, on the project you are mentioned as a main mentor. Can you give me the chanels where I can find help please

avatar Sandra97
Sandra97 - comment - 10 Mar 2019

@Oracions2410 So that we can add you to our communication channel, please drop us your email on our mailing list https://groups.google.com/forum/#!forum/jgsoc

avatar anibalsanchez
anibalsanchez - comment - 14 Mar 2019

Preparing the talk for the Mastermind Podcast, these are my notes about Web Services quick testing 101:

  • Step 1: Download and Install Joomla 4 Alpha 7
  • Step 2: Notice that there two new plugin groups: api-authentication and webservices
  • Step 3: Notice that there two new plugins: API Authentication - Basic Auth and Web Services - Content
  • Step 4: J4 A7 has API Authentication based on Http Basic Authentication. So, yes, you have to configure .htpasswd to test the API. Ref: http://httpd.apache.org/docs/2.4/programs/htpasswd.html
  • Step 5: Of course, if you are a little bit lazy, you can go straight to plugins/api-authentication/basic/basic.php, and change the authentication or create your own plugin.
  • Step 6: Create an article and access the REST API for the Content component. For instance: http://mytest.com/api/index.php/v1/article/1

For more information:

avatar wilsonge
wilsonge - comment - 14 Mar 2019

@Oracions2410 I am - but i was preparing for a presentation i was giving in paris so I wasn't really available. I'm now back up and running. either way however you needed to be added into glip :)

avatar wilsonge
wilsonge - comment - 14 Mar 2019

@anibalsanchez i don't think you need to configure .htpasswd to test the api? i certainly don't need to locally. it's using the joomla user accounts username and password. apache shouldn't need to be aware of them as far as i know

avatar anibalsanchez
anibalsanchez - comment - 14 Mar 2019

In my tests, it's been giving me Forbidden errors until I configured the auth.

Carlos has a similar experience.

avatar carcam
carcam - comment - 14 Mar 2019

Thanks everyone for this awesome feature!!

I can confirm the server doesn't ask me for Username and Password as it's supposed to. I'm using a Lando box using Anibal Lando Recipe for Joomla 4 so I guess this has to do with the server configuration as stated in some comments I found on the internet about this auth method not working out of the box in CGI servers.

In any case I would point out just 2 issues:

  1. with some server configurations you cannot authenticate out of the box and also
  2. Probably checking if user has an account for public content should avoided. I understand the idea of this is to be a "content leak stopper" but we can do better ;)

In any case, again, I was really excited with having this feature when you announced it and I event made a whole episode of my podcast about it because it's awesome to keep creating great solutions for our community and our clients.

avatar wilsonge
wilsonge - comment - 14 Mar 2019

I can confirm the server doesn't ask me for Username and Password as it's supposed to

Wait are you making requests in the browser or something? Nothing is supposed to prompt you for a password?

avatar anibalsanchez
anibalsanchez - comment - 14 Mar 2019

I already told him that he has to configure the .htpassword file to enable the Apache Http Basic Authentication.

I agree with Carlos that if the plugin is enabled, then public content (articles) must be allowed to the Guest user.

Add a Comment

Login with GitHub to post a comment