? Language Change PR-4.0-dev RTC ?

Pending

User tests: Successful: Unsuccessful:

avatar nikosdion
nikosdion
8 Nov 2019

Pull Request for Issue #26925. Closes gh-26925.

Summary of Changes

This PR implements token authentication for the Joomla API application. This is a much more secure approach as already discussed in length; see gh-26925

Testing Instructions

0. Initialization

Install a new site with this PR's branch. If applying on an existing site you need to also:

  • Go to System, Database and fix the schema.
  • Go to System, Discover and install the "API Authentication - Joomla Token" and "User - Joomla Token" plugins.
  • Go to System, Plugins and enable the "API Authentication - Joomla Token" and "User - Joomla Token" plugins.

1. User interface test

Go to the site's backend, Users, Manage.

Edit your Super User.

Expected: There is a User Token tab with an empty token.

Click on Save.

A token is now visible. Copy your token, we'll need it below.

Note: I know the display of the Token tab is a bloody mess. That's a template issue, not something I can fix in my PR. I am simply using an XML form.

2. API access

Now we need to run a request against the API server with one of the following alternatives. In all
cases we assume that the site's URL is http://localhost/joomla4 and the token c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl.

Alternative A. cURL

From the command line:

curl -H "Authorization: Bearer c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl" http://localhost/joomla4/api/index.php/v1/users

Alternative B. WGet

From the command line:

wget --header="Authorization: Bearer c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl" -O - http://localhost/joomla4/api/index.php/v1/users

Alternative C. phpStorm

File, New Scratch File, HTTP Request. Enter the following at the end of the document:

GET http://localhost/joomla4/api/index.php/v1/users
Authorization: Bearer c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl
User-Agent: JoomlaDevelopment/4.0.0
Accept: application/vnd.api+json

###

Choose Run, Run scratch#whatever to execute the HTTP request. CTRL-click (CMD-click on macOS) on the filename appearing above the ### marker to view the response.

Expected Response

A JSON document listing your site's users.

Technical notes

For security reasons we are NOT storing the raw token in Joomla's database. If we were to do that we'd be a SQL injection away from full system compromise. A SQL injection would allow the attacker to dump the tokens. Since tokens are, by nature, both a username and a password they'd be able to fully compromise the site.

Instead, all we store in the database is a seed value that's currently hardcoded to a length of 32 bytes (256 bits). The seed value is generated with the cryptographically secure random bytes generator of Joomla's Crypt package and stored in the database in base64 encoded format. We store this information in the #__user_profiles table.

The token itself is the base64-encoded representation of a string algorithm:user_id:hmac. Below we discuss the various components, not in the same order present in the token.

The hmac is the main security feature of this token implementation. This is the RFC 2104 HMAC of the seed data using Joomla's site secret (the $secret in configuration.php) as the key. This is a very secure process which avoids the known attacks against hash algorithms such as SHA-1 and SHA-256 (reference).

The algorithm tells us which cryptographic hash function was used to generate the HMAC. Per the security considerations of RFC 6151 we forbid the use of the less secure cryptographic hash functions. In fact, the current implementation only allows SHA-256 and SHA-512 with SHA-256 being the (hardcoded) method used to generate the token displayed to the user. That is to say, SHA-512 is added for forward compatibility.

The user_id is the numeric user ID. This is used to make token validation possible. Note that if you have a token you have full access to the site, including the /v1/users endpoint. So please let's not have the pointless discussion of whether the token is "leaking" the user ID. If you can't understand why it doesn't you're not qualified to discuss token authentication.

The api-authentication plugin expects to see the token in the Authorization HTTP header in the request. We strongly recommend using the HTTP header method. The header MUST have the format:
Authorization: Bearer <token>
where <token> is your Joomla API token in base64 encoding. Note that the string Bearer is case sensitive and must have at least one space separating from the token. Any additional whitespace around the token is discarded.

Upon receiving the token, the api-authentication plugin breaks it into its known parts. As long as the algorithm is one of the supported values (sha256 or sha512) authentication proceeds. We read the user profile information and retrieve the seed data for the user_id in the token message. Using this seed data from the database and the site's secret from configuration.php we calculate the reference HMAC. The reference HMAC is compared to the retrieved HMAC from the token message using a time-safe comparison. It is worth noting that even when we encounter an invalid user ID we still go through all the motions for authentication to prevent timing attacks which would allow user enumeration.

As long as the HMACs match, the token is enabled and the user belongs to the allowed user groups and is not blocked, not yet activated or has requested a password reset we are going to grant them access to the API application.

Since the API application is currently only allowed for Super Users we limit the access to the token management UI and token authentication only to Super Users by default. This is user configurable in the user/token plugin.

For security and privacy reasons users can only view their own token, even if they are Super Users. Users with com_users edit privileges can also disable, enable or reset another user's token BUT they still cannot view it. This is useful in case there is a suspicion that the token of a user was compromised and another responsible adult needs to log into the site's backend and disable or reset it before things get out of hand.

There are two plugins, a user plugin to render the management UI and an api-authentication which performs the actual token validation. Combining them would pose architectural issues and give the wrong ideas to third party developers and users as discussed in gh-26925. A potentially confusing situation would arise when the api-authentication plugin is disabled and the user plugin is enabled: the management UI is there but nobody's home to answer the door. To prevent that the user plugin will refrain from rendering the token management UI when the api-authentication plugin is disabled. The reverse situation DOES NOT cause the api-authentication plugin to disable itself. There are valid reasons to do that, e.g. prevent a nosy client from breaking things while swearing that they didn't touch anything...

Finally, as it should be pretty obvious, the token authentication only works for Joomla's API application, NOT the frontend and backend applications. Moreover, due to the remote and unattended nature of API application access the token authentication bypasses Two Factor Authentication.

As far as security goes, at worst for us and best for them, an attacker would need to guess correctly the HMAC-SHA256 sum of a user on top of their user ID. An HMAC-SHA25 sum is 256 bits long and chaotic in nature. This means that an attacker would need to go through approximately half of the key space before cracking it, i.e. 2^128 requests. That's more than 340,000,000,000,000,000,000,000,000,000,000,000,000 requests to the server. We can expect the heat death of the universe before that happens.

HMAC-SHA256 algorithm attacks are not even in a theoretical stage yet. Besides, a such an attack would require at least two pairs of the seed data and HMACs -- but in this case you'd be already hacked since the attacker already has valid tokens for your site. Disclaimer: I am not a cryptographer so take this with a pinch of salt.

In practical terms, the only way you could conceivably beat the token is capturing the actual token with a man-in-the-middle attack or hacking the user's device or application / service which uses the token. This is an acceptable risk in the context of remote API access.

I will preemptively answer the inevitable question: why not encrypt the token itself in the database. The short answer is that it's not any more secure than HMAC sums. If anything, a SQL injection may give the attacker enough data to attack the encryption algorithm itself (there are known attacks if you know the message size and have enough ciphertext samples).

And the second inevitable question: why not keep the token itself hashed in the database. It's obvious. Because we need to show the token to the user. Storing a hash means we can never get the raw data back and show it to the user. As to why we can't save the raw data encrypted, see above.

Using a calculated HMAC-SHA256 or HMAC-SHA512 from two pieces of information, one in the database and one in a file, gives us better security than other methods while still allowing us to display the token to the user anytime they request it.

Backwards compatibility

While the code in this PR does not break backwards compatibility it does disable the api-authentication/basic plugin by default. This is intentional. The problem we started out solving was that the api-authentication/basic plugin allowed miscreants to brute force the Super User's password and gain access to the API application, therefore pwning (completely compromising) the site. Please DO NOT revert the SQL code that disables the basic authentication plugin, that would be security suicide.

Translation impact

A total of 4 translation files and 12 unique translation strings was added. Expected translation time impact: approximately 15 minutes.

Documentation Changes Required

Someone needs to tidy up the test instructions and technical notes in a coherent documentation page.

Updates to this PR

Some things changed since I submitted this PR. Noting here so the context of the comments below does make sense:

  • Fixed the HTTP header name from Authentication (WRONG!) to Authorization. My bad.
  • Removed the _authToken query string parameter. It is not actually necessary. Even Apple's Shortcuts app allows for custom HTTP headers.
avatar nikosdion nikosdion - open - 8 Nov 2019
avatar nikosdion nikosdion - change - 8 Nov 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 8 Nov 2019
Category SQL Administration com_admin Postgresql Language & Strings Installation Front End Plugins
avatar SharkyKZ
SharkyKZ - comment - 8 Nov 2019

Please DO NOT revert the SQL code that disables the basic authentication plugin, that would be security suicide.

If Basic plugin is so insecure, why not just remove it?

avatar nikosdion
nikosdion - comment - 8 Nov 2019

@SharkyKZ It's not my decision to remove it, it's something @wilsonge has to decide. I assume that once the token authentication is merged we can revisit that.

avatar nikosdion nikosdion - change - 8 Nov 2019
Labels Added: Language Change PR-4.0-dev
avatar brianteeman
brianteeman - comment - 8 Nov 2019

testing 1. User interface test

Instead of the label being reset how about calling it Create/Reset To me this would be more obvious when creating a token for the first time.
OR
Have two conditional fields
Create (if no token exists)
Reset (if token exists)

Secondly if we add an onchange event then it will be saved automatically - much more obvious to me and removes the need for the lengthy description.

I can supply a pull request to your branch if you agree

avatar nikosdion
nikosdion - comment - 8 Nov 2019

It's called Reset because it's only used to reset an existing token. It is not to create a token. A token is created automatically the first time you save your user profile without having a token already created.

I am not sure what onchange event you have in mind and I can't see how JavaScript would help when what you need is submit a form which might be in the frontend or the backend or rendered inside a third party user profile management component.

The only other alternative is magically creating a token for everyone editing their user profile. I don't like that approach. Besides, the only people who actually need an API token are power users who should have the presence of mind to read text on the screen and carry out a simple instruction like "save and come back here". If we can't trust our power users to do that let's just delete the entire 4.0 branch and relaunch Mambo 4.0 in its stead.

avatar brianteeman
brianteeman - comment - 8 Nov 2019

A token is created automatically the first time you save your user profile without having a token already created

I had no way of knowing that :)

avatar brianteeman
brianteeman - comment - 8 Nov 2019

For security and privacy reasons users can only view their own token, even if they are Super Users.

I agree with this but as it just caught me out then I think there should be an on screen message for this

avatar brianteeman
brianteeman - comment - 8 Nov 2019

Finally I can not get any further with testing as using curl from the cli I get a response of
{"errors":{"code":500,"title":"Internal server error"}}

And using Bearer with postman I get
{"errors":[{"title":"Forbidden"}]}

avatar brianteeman
brianteeman - comment - 8 Nov 2019

It's called Reset because it's only used to reset an existing token. It is not to create a token. A token is created automatically the first time you save your user profile without having a token already created.

That absolutely makes sense when you are creating a new user after you have enabled this plugin. The problem is when you add the plugin to an existing site and then open an existing user for editing.

avatar brianteeman
brianteeman - comment - 8 Nov 2019

When you try to block a user from the user list (or enable a blocked user) you now get the following error
Argument 4 passed to PlgUserToken::onUserAfterSave() must be of the type string, null given, called in C:\htdocs\joomla-cms\libraries\src\Plugin\CMSPlugin.php on line 285

avatar nikosdion
nikosdion - comment - 8 Nov 2019

That absolutely makes sense when you are creating a new user after you have enabled this plugin. The problem is when you add the plugin to an existing site and then open an existing user for editing.

Yes, I can fix that but it will cost me half a day. Is it worth it?

When you try to block a user from the user list (or enable a blocked user) you now get the following error

Thanks. I'll fix that. The problem is Joomla documents sod all about its events. Following the kinda-sorta documentation of docblocks of existing plugins and transcribing them to PHP 7 parameter types results in these kinds of issues :/

avatar nikosdion
nikosdion - comment - 8 Nov 2019

I agree with this but as it just caught me out then I think there should be an on screen message for this

That's another half day but, hey, if you think it's worth it.

Finally I can not get any further with testing as using curl from the cli I get a response of

That's probably something unrelated. I cannot reproduce it.

And using Bearer with postman I get

I don't know what postman is. I can only comment on using cURL, wget and phpStorm i.e. two standard, cross-platform CLI tools and the IDE used by the Joomla community developers.

avatar brianteeman
brianteeman - comment - 8 Nov 2019

I agree with this but as it just caught me out then I think there should be an on screen message for this

That's another half day but, hey, if you think it's worth it.

come on - its about 30 minutes maximum - even for me.

Postman == https://getPostman.org
Its the number one tool for testing and building api such as this

avatar brianteeman
brianteeman - comment - 8 Nov 2019

oops - typo - its https://getpostman.com

avatar wilsonge
wilsonge - comment - 8 Nov 2019

@SharkyKZ It's not my decision to remove it, it's something @wilsonge has to decide. I assume that once the token authentication is merged we can revisit that.

I'm happy to bin basic auth. It was something that allowed me to got an auth implementation working in the least amount of time possible. It wasn't to stick around long term if something better came along

I'm busy in the office today. And have a full day tomorrow outside of Joomla so I'll look into code review on Sunday and the full testing instructions you've written up.

Thankyou so much for working on this!

avatar brianteeman
brianteeman - comment - 8 Nov 2019

When you are the user
image

When you are not the user
image

Line12 add
use Joomla\CMS\Language\Text;
Line 74 add
echo "<div class='alert alert-info'>" . Text::_('PLG_USER_TOKEN_HIDDEN_NOT_USER') . "</div>";

And then add the language string

avatar nikosdion
nikosdion - comment - 8 Nov 2019

See latest commit.

avatar brianteeman
brianteeman - comment - 8 Nov 2019

Thank you

avatar nikosdion
nikosdion - comment - 8 Nov 2019

Adding alert classes to an input field is dirty and will probably no longer work in the future. These classes should only really be added to presentation elements, not input elements. What I'm doing is use three note fields, each one being displayed under different conditions. It makes the code much more unreadable but at least you get your correct message each time.

avatar brianteeman
brianteeman - comment - 8 Nov 2019

Perhaps my method is better ? No idea.

avatar nikosdion
nikosdion - comment - 8 Nov 2019

No, your method is the dirty one. The note fields are the cleaner method. If at some point we need to change the container element and / or the class we can simply edit the XML file.

Regarding Postman, I am not comfortable creating a login to cloud sync my request data considering that they contain my tokens. I want to keep my secrets, well, secret. The standard tool for testing APIs is cURL. Do you remember Lorna Jane's presentation from a few years ago? For fancier presentation of the JSON data phpStorm works perfectly and keeps all my data on my own encrypted drive, thank you very much.

I believe that you must have made a mistake either copying the token or with the query string parameter. Here is a real world example from my Joomla 4 core development site:

GET http://jdev4.local.web/api/index.php/v1/users?_authToken=c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl
User-Agent: JoomlaDevelopment/4.0.0
Accept: */*
{
	"links": {
		"self": "http:\/\/jdev4.local.web\/api\/index.php\/v1\/users?_authToken=c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl",
		"first": "http:\/\/jdev4.local.web\/api\/index.php\/v1\/users?_authToken=c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl&page[offset]=0&page[limit]=20",
		"next": "http:\/\/jdev4.local.web\/api\/index.php\/v1\/users?_authToken=c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl&page[offset]=20&page[limit]=20",
		"previous": "http:\/\/jdev4.local.web\/api\/index.php\/v1\/users?_authToken=c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl&page[offset]=0&page[limit]=20",
		"last": "http:\/\/jdev4.local.web\/api\/index.php\/v1\/users?_authToken=c2hhMjU2Ojk3MTpiM2IzYTZjNzRmNGUwY2U0NGM3OGVkMDcyMjdhYjFlOWFmMzExODExZjZhNmE3ZGFlZDUwZWRiNmJkZTVlYzJl&page[offset]=0&page[limit]=20"
	},
	"data": [
		{
			"type": "users",
			"id": "972",
			"attributes": {
				"id": 972,
				"name": "Foo Bar",
				"username": "foobar",
				"email": "REDACTED :)",
				"group_count": 1,
				"group_names": "Super Users"
			}
		},
		{
			"type": "users",
			"id": "971",
			"attributes": {
				"id": 971,
				"name": "REDACTED :)",
				"username": "REDACTED :)",
				"email": "REDACTED :)",
				"group_count": 1,
				"group_names": "Super Users"
			}
		}
	],
	"meta": {
		"total-pages": 1
	}
}
avatar nikosdion
nikosdion - comment - 8 Nov 2019

One thing I did observe is that randomly Accept: application/vnd.api+json did not work and when I retried worked fine (huh?!) but Accept: */* always worked. Moreover, Accept: application/json i.e. the correct MIME type per RFC 4629 never works with error "Could not match accept header". This is a bug in the API application, not this PR. It's out of scope so I'm not touching it :)

avatar brianteeman
brianteeman - comment - 8 Nov 2019

You do not need to sign in ;)

Need to know more - I can show you next week or check out the videos of @vdespa - I am sure you remember him

avatar brianteeman
brianteeman - comment - 8 Nov 2019

grrrrh

avatar brianteeman
brianteeman - comment - 8 Nov 2019

Same 500 error with linux
curl -H "Authentication: Bearer c2hhMjU2OjYzMDowOTI3MzQzYTEyNDUwZDliZDZhN2Q3N2IwNzY3N2JlNTUwOWY2N2IyMzZlNzUyODY5NGNkNjQyYzQwMzg1NzNh" http://localhost/joomla-cms/api/index.php/v1/users

avatar nikosdion
nikosdion - comment - 8 Nov 2019

Which version of PHP are you using?

Do you have the PHP hash module enabled? Check the phpinfo() output. While doing that also check that the "Hashing Engines" include sha256 and sha512.

avatar brianteeman
brianteeman - comment - 8 Nov 2019

PHP Version 7.2.17
hash support	enabled
Hashing Engines	md2 md4 md5 sha1 sha224 sha256 sha384 sha512/224 sha512/256 sha512 sha3-224 sha3-256 sha3-384 sha3-512 ripemd128 ripemd160 ripemd256 ripemd320 whirlpool tiger128, 3 tiger160, 3 tiger192, 3 tiger128, 4 tiger160, 4 tiger192, 4 snefru snefru256 gost gost-crypto adler32 crc32 crc32b fnv132 fnv1a32 fnv164 fnv1a64 joaat haval128, 3 haval160, 3 haval192, 3 haval224, 3 haval256, 3 haval128, 4 haval160, 4 haval192, 4 haval224, 4 haval256, 4 haval128, 5 haval160, 5 haval192, 5 haval224, 5 haval256, 5
avatar nikosdion
nikosdion - comment - 8 Nov 2019

You'll have to check your PHP error log and see what the 500 error is about. I cannot reproduce it and it's not coming from trying to calculate the HMAC. It's also impossible that you disabled base64_encode/_decode because the UI would also fail. So you're probably hitting some other Joomla core issue.

avatar wilsonge
wilsonge - comment - 8 Nov 2019

One thing I did observe is that randomly Accept: application/vnd.api+json did not work and when I retried worked fine (huh?!) but Accept: / always worked. Moreover, Accept: application/json i.e. the correct MIME type per RFC 4629 never works with error "Could not match accept header". This is a bug in the API application, not this PR. It's out of scope so I'm not touching it :)

So parts of this is expected. Accept: application/vnd.api+json clearly should work. I've not seen a one off blip before (neither seen it in the (admittedly limited) drone api tests) so unless you see it again or can reproduce it I don't really have anything to go on

Accept: / always worked

This is expected because obviously it should match anything and the priority for all our api endpoints is JSON API (app default here https://github.com/joomla/joomla-cms/blob/4.0-dev/libraries/src/Application/ApiApplication.php#L226 and can be overridden at the endpoint plugin level).

Moreover, Accept: application/json i.e. the correct MIME type per RFC 4629 never works with error

This is also expected. I mapped JSON to the existing Joomla JSON format rather than to JSON API. No API endpoints currently utilise that format https://github.com/joomla/joomla-cms/blob/4.0-dev/libraries/src/Application/ApiApplication.php#L78 so it's correctly returns back no matching content type. It can get complicated because if you support core JSON API and you write a custom endpoint for say JSON LD then which should JSON default return? So I left it to it's own Joomla format that people can proxy with plugins as required. I understand it's maybe not the most expected thing ever but it's not totally invalid.

avatar nikosdion
nikosdion - comment - 9 Nov 2019

@wilsonge This is a fair point. However, I am mostly concerned about how it's going to affect compatibility with dumber client software that ask for application/json. I am thinking Apple's Shortcuts app which I think does that (I'll have to check again). Same for other automation GUIs / services where the user doesn't have full control of the request. I guess in those cases I should write a system plugin which converts the Accept header? (Sorry for the off-topic)

avatar nikosdion nikosdion - change - 10 Nov 2019
The description was changed
avatar nikosdion nikosdion - edited - 10 Nov 2019
avatar nikosdion
nikosdion - comment - 10 Nov 2019

I had mistyped the header. It should be Authorization: Bearer <token>. I corrected the code and the testing instructions.

avatar brianteeman
brianteeman - comment - 10 Nov 2019

After fixing that and another array issue I can confirm that the current code now works great!!!

avatar brianteeman
brianteeman - comment - 10 Nov 2019

(the drone errors are valid but are unrelated to this PR)

avatar brianteeman brianteeman - test_item - 10 Nov 2019 - Tested successfully
avatar brianteeman
brianteeman - comment - 10 Nov 2019

I have tested this item successfully on cc42da3


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/27021.

avatar wilsonge
wilsonge - comment - 10 Nov 2019

I am thinking Apple's Shortcuts app which I think does that (I'll have to check again). Same for other automation GUIs / services where the user doesn't have full control of the request. I guess in those cases I should write a system plugin which converts the Accept header? (Sorry for the off-topic)

Yeah I mean there's deliberate scope to override that with the addFormatMap in plugins (even webservice plugins in theory should be able to override that as their assembling routes - whether that's a good idea or not - technically it's possible).

I mean I'm not wedded to the concept - if it turns out it's easier to map to json api for concrete reasons I'm happy to revisit that decision. But it's not some accident/coding bug - and it's easy to change in one line if we do go to change it

avatar nikosdion nikosdion - change - 10 Nov 2019
The description was changed
avatar nikosdion nikosdion - edited - 10 Nov 2019
avatar joomla-cms-bot joomla-cms-bot - change - 11 Nov 2019
Category SQL Administration com_admin Postgresql Language & Strings Installation Front End Plugins SQL Administration com_admin Postgresql Language & Strings Installation Libraries Front End Plugins
avatar wilsonge
wilsonge - comment - 11 Nov 2019

Think this is looking good. @brianteeman are you able to do a last test today. And I'll go test on my local setup tonight and get this merged.

avatar brianteeman
brianteeman - comment - 11 Nov 2019

As far as the scope of this PR is concerned I am happy with how it works

avatar wilsonge
wilsonge - comment - 11 Nov 2019

The Active status for the token (called enabled in the code) doesn't seem to be disabled. If on a profile I try and disable the field and hit save nothing is updated in the db and the field refreshes back to active. if we fix that should be good to go I think

I'm in two minds about how I feel about the API Auth having it's own user groups in the UI compared to general Joomla usage. Part of that is trying to evaluate how many people are going to have different Auth types based on different user groups who both have API access

Don't think it's something to fix - but more to document: One thing that caught me out was without the htaccess file being placed properly - everything fell apart :) Specifically this line missing https://github.com/joomla/joomla-cms/blob/staging/htaccess.txt#L73 meant I never received the auth header and got forbidden errors forever. It appears to be a known thing so don't think there's a workaround - Ref: https://stackoverflow.com/questions/26475885/authorization-header-missing-in-php-post-request

avatar wilsonge
wilsonge - comment - 11 Nov 2019

I've also updated the authentication specification here to reflect what's been implemented https://docs.joomla.org/Joomla_Api_Specification

avatar wilsonge
wilsonge - comment - 11 Nov 2019

Final thing is how to deal with the codeception api tests (the drone failures here). I can probably force the token generation in the UI. But I dunno if I have a way of taking the value generated and pumping it into the tests (e.g. https://github.com/joomla/joomla-cms/blob/4.0-dev/tests/Codeception/api/com_content/ContentCest.php#L60 )

avatar nikosdion
nikosdion - comment - 12 Nov 2019

@wilsonge The enabled not saving was a symptom of the additional XML form not being loaded on save. I fixed this.

Regarding the Authorization header, I made a small change. If the Authorization header is not available to PHP we can pass the HTTP header X-Joomla-Token: your_token_here instead which does work without the .htaccess file. You can update the API specification.

Regarding the tests, you can always write a known string to the #__user_profiles table for a known user and a known secret in configuration.php. Therefore the token will be known. Even if you can't change the secret you can still calculate the token, it's a simple HMAC-SHA256. You can safely do that in the _before() method of the test, I guess.

avatar alikon
alikon - comment - 13 Nov 2019

i'm experiencing issues, i've tested with curl,wget, postman
Screenshot from 2019-11-13 19-21-19

and yes i've hash support enabled on php 7.3.11
Screenshot from 2019-11-13 19-29-16

avatar nikosdion
nikosdion - comment - 13 Nov 2019

@alikon As @wilsonge already pointed out, if you do not use Joomla's default .htaccess the Authorization header will not work at all.

Try replacing your "Authorization: Bearer " with "X-Joomla-Token: " (watch out! No Bearer in the latter form!) OR use Joomla's default .htaccess file to fix the Apache / PHP issue with the Authorization header.

avatar alikon
alikon - comment - 13 Nov 2019

oops, sorry i've missed that .... and yes i'm not using the

Joomla's default .htaccess

let me re-test ....

avatar alikon
alikon - comment - 13 Nov 2019

yes right, now works as expected ....maybe we need to add this to the documentation to avoid silly people like me to do silly mistake 😄
Screenshot from 2019-11-13 20-02-01

avatar alikon alikon - test_item - 13 Nov 2019 - Tested successfully
avatar alikon
alikon - comment - 13 Nov 2019

I have tested this item successfully on 89dcfd1


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/27021.

avatar wilsonge
wilsonge - comment - 14 Nov 2019

Apologies been feeling ill the last few days so haven't had time to look into the tests yet - proposition you made sounds viable just need to feel good enough to sit down and try some things out.

avatar anibalsanchez anibalsanchez - test_item - 15 Nov 2019 - Tested successfully
avatar anibalsanchez
anibalsanchez - comment - 15 Nov 2019

I have tested this item successfully on 89dcfd1

From #JMAD19 PBF, Test OK

With the .htaccess, all commands are working OK:

curl -H "Authorization: Bearer c2hhMjU2OjY3NzoxOWU1YmRlYTEwMGQ4ZDkzODU5NDhkNDFkZTM0YjMzYWNlODM4OGEyMjdkZjExNmU1MjVkZjc1OTliN2Q4NDEw" http://j4.lndo.site/api/index.php/v1/users

wget --header="Authorization: Bearer c2hhMjU2OjY3NzoxOWU1YmRlYTEwMGQ4ZDkzODU5NDhkNDFkZTM0YjMzYWNlODM4OGEyMjdkZjExNmU1MjVkZjc1OTliN2Q4NDEw" -O - http://j4.lndo.site/api/index.php/v1/users

curl -H "X-Joomla-Token: c2hhMjU2OjY3NzoxOWU1YmRlYTEwMGQ4ZDkzODU5NDhkNDFkZTM0YjMzYWNlODM4OGEyMjdkZjExNmU1MjVkZjc1OTliN2Q4NDEw" http://j4.lndo.site/api/index.php/v1/users

This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/27021.
avatar alikon alikon - change - 15 Nov 2019
Status Pending Ready to Commit
avatar alikon
alikon - comment - 15 Nov 2019

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/27021.

avatar wilsonge
wilsonge - comment - 27 Nov 2019

@Hackwar have you had any time to look into the tests here like we discussed?

avatar rdeutz
rdeutz - comment - 3 Feb 2020

This has to pass the testing, I would have looked into this but the build is no longer available, @nikosdion can you just push some change to a file, empty line is enough to trigger a build. I can't restart or look into a non existing build.

avatar wilsonge
wilsonge - comment - 3 Feb 2020

@rdeutz it's pretty simple. The api tests need to generate a token of the format nik is building here. the easiest way is to force the secret of the dummy site to a known value to build it. harder would be after the install is done to read it out from configuration.php. Either is valid.

avatar rdeutz
rdeutz - comment - 3 Feb 2020

@wilsonge I am not talking about code, I want to have a build running so I can have a look. The last build is too old and have beed deleted from the history.

avatar nikosdion nikosdion - change - 4 Feb 2020
Labels Added: RTC
avatar nikosdion
nikosdion - comment - 4 Feb 2020

@rdeutz Okey dokey!

avatar rdeutz
rdeutz - comment - 6 Feb 2020

@nikosdion sorry for being a pain but I don't get the testing run, can you update (rebase) your branch with 4.0-dev, your PR is too far away from the latest commit and we have set the clone step only to go 42 levels deep back. I changed this in the meantime but your PR uses the old settings. Thanks.

avatar nikosdion nikosdion - change - 8 Feb 2020
Labels Added: ?
avatar nikosdion
nikosdion - comment - 8 Feb 2020

Oh, wow, saying that the branch was out of date was an understatement and a half. It was 680 commits behind 💀 I updated it and hopefully this triggers the build.

FWIW merging was far easier than rebasing considering a. how far the branches had diverged and b. the fact that the vast majority of changes I made were about adding a new plugin.

avatar richard67
richard67 - comment - 9 Feb 2020

Restarted Drone just to see what happens ;-)

avatar richard67
richard67 - comment - 9 Feb 2020

Well, as before: API tests failing, rest ok. So it seems the next step is to adapt the API tests to this PR as @wilsonge described in his comment above.

@rdeutz Who can do that? How to continue with this PR here?

avatar richard67
richard67 - comment - 15 Mar 2020

Ping @wilsonge @rdeutz : How does it go on with fixing API tests so we can merge this PR?

avatar wilsonge
wilsonge - comment - 15 Mar 2020

Unsure - I think Robert prioritised a couple of other beta blockers on this one (the b/c break in the model class now fixed etc). But it's assigned with him at the moment

avatar wilsonge
wilsonge - comment - 22 Mar 2020

@nikosdion nikosdion#5 here's a PR to fix tests. @rdeutz do you wanna take a look too?

avatar nikosdion nikosdion - change - 23 Mar 2020
Labels Added: ?
Removed: ?
avatar joomla-cms-bot joomla-cms-bot - change - 23 Mar 2020
Category SQL Administration com_admin Postgresql Language & Strings Installation Front End Plugins Libraries Repository SQL Administration com_admin Postgresql Language & Strings Installation Libraries Front End Plugins Unit Tests
avatar nikosdion nikosdion - change - 23 Mar 2020
Labels Added: ?
avatar nikosdion
nikosdion - comment - 23 Mar 2020

I have rebased the PR to the 4.0-dev branch and merged George's PR. 🤞

avatar wilsonge
wilsonge - comment - 23 Mar 2020

Always helps if i properly port my fixes from my dev branch into the commits :) nikosdion#6

avatar nikosdion nikosdion - change - 23 Mar 2020
Labels Added: ?
Removed: ?
avatar wilsonge
wilsonge - comment - 23 Mar 2020

Attempt 3 nikosdion#7

avatar wilsonge
wilsonge - comment - 23 Mar 2020

Just to confirm ran another test #28441 and drone passes with this final change

avatar wilsonge wilsonge - change - 23 Mar 2020
Status Ready to Commit Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2020-03-23 21:04:29
Closed_By wilsonge
Labels Added: ?
Removed: ?
avatar wilsonge wilsonge - close - 23 Mar 2020
avatar wilsonge wilsonge - merge - 23 Mar 2020
avatar wilsonge
wilsonge - comment - 23 Mar 2020

Tested this locally. Everything seems to work and as I'm happy I've got the tests working I'm merging this.

Thankyou very much for all your perseverance here @nikosdion !

avatar nikosdion
nikosdion - comment - 24 Mar 2020

Woohoo! This merge makes me VERY happy. It is a feature I need in my own software.

I was going to release the token management user plugin myself. Now that Joomla 4 has it built in I can limit my plugin's scope to Joomla! 3 and use the native 4.0 functionality. Yay, standardization!

Add a Comment

Login with GitHub to post a comment