? ? ? ? Pending

User tests: Successful: Unsuccessful:

avatar nikosdion
nikosdion
20 Oct 2019

Pull Request for Issue #26583 .

This PR replaces #26727. The feature branch got borked in the old PR. Furthermore, the new PR also includes the stretch goal of removing updates with the old / missing Download Key when you save the Download Key again.

Summary of Changes

This PR improves the user experience for the Download Key support in Joomla 4.

  • Quickicon plugin to display whether any Download Keys are missing or if they are all entered correctly. The icon links to the Update Sites, filtered by those in need of / supporting Update Sites respectively. If no update site requires a Download Key the icon is not even shown.
  • The Update Sites page lets you filter by Update Sites which support Download Keys; or those which are missing a Download Key; or those who have a valid Download Key.
  • The Update Sites page clearly marks extensions which support the Download Key feature but do not have a Download Key entered.
  • When you edit an Update Site, the Download Key field only displays for extensions which support it (those that have a dlid attribute in their XML manifest file).
  • The update page does not allow you to update extensions which need a Download Key but no Download Key is entered in their update site.

Testing Instructions

First, install this fake "paid download" extension: Null Files 1.0. This is a fake extension which creates a .txt folder in your site's temp-folder and, crucially, creates an update site that requires a Download Key to work. Without the download key it returns a 403 error, just like paid extensions.

Don't worry, no payment is necessary for this fake extension. The Download Key for this extension is PizzaBugsAndFun2019.

Since we're testing lots of things I have split the testing in 6 sections with one or more test results. Each desirable test result is marked as "Checkpoint 1.1", "Checkpoint 2.1" etc. If your test fails please indicate which checkpoint you were stuck at.

You need to follow the test steps in the order specified. If you get lost at any point please uninstall the Null Files extension, reinstall it and start over.

Before installing the patch

1. Affirming the existence of UX problems

  • Go to your site's administrator control panel.
  • Checkpoint 1.1 There is no indication you have installed a paid extension in need of a Download Key is necessary.
  • Go to System, Update Sites.
  • Click on any Joomla Core.
  • Checkpoint 1.2 You should see a Download Key field. This verifies that a UX problem exists for free extensions (Joomla misleads us a Download Key is required)
  • Go to System, Update.
  • Try to install the update to our fake extension, “Null Files”.
  • Checkpoint 1.3 It fails with a puzzling 403 message. Note: you might NOT see the message because of an unrelated user interface bug in the template.

After installing the patch.

2. Verify that the Control Panel instructs you to enter the Download Key*

  • Go to your site's administrator control panel.
  • Checkpoint 2.1 You will see the message "A Download Key is missing!" message under the Update Checks area.
  • Click on the icon above the "A Download Key is missing!" message.
  • Checkpoint 2.2 You should now be in the Update Sites page, showing only the “Null Files” extension

3. Verify that Download Key filtering works*

  • Click on the “Null Files” extension.
  • Enter the Download Key (PizzaBugsAndFun2019) and click on Save & Close.
  • Checkpoint 3.1 The list no longer displays anything
  • From the Filter Options dropdown change "Download Key invalid" to "Download key valid".
  • Checkpoint 3.2 You should see the Null Files extension and the Download Key you entered before
  • From the Filter Options dropdown change "Download Key valid" to "Download key" (the first option).
  • Checkpoint 3.3 You should see all Update Sites now
  • Click on the “Null Files” item again and remove the Download Key
  • Click on Save & Close
  • Checkpoint 3.4 The Null Files update site now shows again with a warning that a Download Key is required

4. Verify that non-paid extensions do NOT ask you for a Download Key

  • Click on the Joomla! Core update site
  • Checkpoint 4.1 You do not see a Download Key field
  • Click on Close

5. Verify that updates are disallowed when the Download Key is missing

  • Go to System, Update
  • Checkpoint 5.1 There is an update available for “Null Files”. However, you see a ban icon instead of a checkbox to its left AND a message that the Download Key is missing.

6. Verify that updates do work with a valid Download Key

  • Go back to your administrator control panel page
  • Click again on the "A Download Key is missing!" message
  • Click on Null Files again
  • Enter the Download Key PizzaBugsAndFun2019
  • Click on Save & Close
  • Go to System, Update
  • Checkpoint 6.1 There is NO update available for “Null Files”. This is expected! When you save a new Download Key we need to delete old updates, otherwise Joomla wouldn't be applying the new Download Key (it's a long story why it happens and it dates back over 10 years ago).
  • Click on Find Updates
  • Checkpoint 6.2 You should now be able to select the “Null Files” update and click on Update
  • The update is installed

Documentation Changes Required

I assume the help pages for the backend control panel, the Update Sites and the Update pages need to be amended.

Special thanks

Many thanks go to the Greek Joomla! community for organizing a successful Pizza, Bugs and Fun 2019 event. This PR was
written during PBF19.

Special thanks to my wife, @crystalenka. Most of the ideas in this PR and the understanding of how users think about
Download Keys are based on her UX improvement work for my company's software.

Note about rebasing the branch

@HLeithner This branch is safe for rebasing to 4.0-dev. If it breaks I have a backup. However, I would appreciate it if you'd first try to rebase / merge locally, make sure you can actually log into the backend of the site and access the Update Sites page before pushing the corresponding button on GitHub. I want to spend my time fixing bugs in the software, not the repo mess caused by an opaque operation hidden behind a button.

avatar nikosdion nikosdion - open - 20 Oct 2019
avatar nikosdion nikosdion - change - 20 Oct 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 20 Oct 2019
Category Administration com_installer Language & Strings SQL Installation Postgresql Libraries External Library Composer Change Front End Plugins
avatar brianteeman
brianteeman - comment - 20 Oct 2019

Did you mean to remove libraries/vendor/.htaccess ?

avatar HLeithner
HLeithner - comment - 20 Oct 2019

@nikosdion i will not touch you pr, thx for continuing

avatar HLeithner
HLeithner - comment - 20 Oct 2019

Btw. Yiu have a checkbox when creating the PR it can be changed by maintainers.

avatar nikosdion
nikosdion - comment - 21 Oct 2019

@brianteeman No, I didn't. The vendor folder is managed by Composer and is meant to be possible to delete completely (which is what I did when Composer barfed installing requirements). The correct approach would be adding a .htaccess and a web.config file in the libraries folder but I digress. I'll add the file back to the wrong folder.

@HLeithner The checkbox placing and text makes it sound like it controls whether maintainers can edit the PR description, not have access to my repo. That's terrible UX in GitHub. Microsoft took over and it shows. I'll now know to blindly check a box with a misleading label. Thank you!

avatar nikosdion nikosdion - change - 22 Oct 2019
Labels Added: ? ? ?
avatar nikosdion
nikosdion - comment - 22 Oct 2019

@brianteeman I forgot to comment again about the .htaccess issue after looking at the actual directory tree. I can tell you that the .htaccess file in libraries/vendor is unneeded and removing it is correct. The parent folder does already implement the correct solution (using a .htaccess and web.config file to block all web access to libraries and all its subdirectories). So I'm not changing this again since it would ultimately be the wrong thing to do.

So, as far as I'm concerned this PR is ready to be fully tested.

avatar brianteeman
brianteeman - comment - 22 Oct 2019

Did you mean to remove libraries/vendor/.htaccess ?

I guess the answer then is yes you did mean to remove it without even realising it ;)

avatar nikosdion
nikosdion - comment - 22 Oct 2019

Yes :) I did what any sane developer would do (remove the vendor folder before retrying to install Composer dependencies). So by virtue of doing something reasonable and expected I corrected a weird and unnecessary implementation detail in Joomla :D

avatar brianteeman
brianteeman - comment - 22 Oct 2019

it looks like a file that was simply overlooked when the switch to composer was made

avatar Quy
Quy - comment - 22 Oct 2019

Remove the Save button for non-paid extensions.

I wanted to link to the Download Key page but the template's CSS was betraying me.

Try this <a href="#" class="badge badge-warning">Warning</a>

avatar nikosdion
nikosdion - comment - 23 Oct 2019

@Quy I will not remove the Save button for free extensions for two reasons:

  1. It clears any lingering Download Key. Think about downgrading a paid extension to its free edition with the developer simply changing the Update Site's title and URL.
  2. Saving an Update Site now also resets the last update check time which is of paramount importance when Joomla's updates are stuck. This happens seldom but when it does there's currently no other way to fix it than manual database surgery (or using regenerate update sites which cocks everything up and removes all download keys in the process). After this PR is merged you can tell people to simply click on the update site of the extension they are no longer receiving updates, save and then head back to Updates and click its refresh button.

As for the warning, in the process I realized that you may have multiple paid extensions missing a Download Key. The correct behavior is a warning at the top of the page (system warning) with a link to the Update Sites page filtered by extensions with missing Download Keys. This is what I already did.

avatar nikosdion
nikosdion - comment - 23 Oct 2019

I have implemented all suggestions from @brianteeman, @alikon and @Quy feedback. Please test the PR so I can subsequently work on bringing it up-to-date with the 4.0-dev branch. Thank you!

avatar brianteeman
brianteeman - comment - 23 Oct 2019

Is there a scenario where you would have a download key for an extension that isn't paid for? If so then we need to tweak the terminology.

avatar nikosdion
nikosdion - comment - 23 Oct 2019

Great question, @brianteeman. In theory nothing prevents a developer from requiring a Download Key for a free of charge extension. However, in the last 15 years Joomla's been around I have not seen any free extension which limits its downloads. I can't even think of why would anyone do that. So I prefer to add language implying that the extension is "paid" or "for a fee" rather than ambiguous language such as "limited distribution" or something to that effect.

If we wanted to be very precise here we could let the developer provide a language key to override or be appended to the default message, ideally giving better instructions to their clients about the Download Key (e.g. "Updates to Foobar require an active, paid subscription to Acme Corp. This is checked by means of the Download Key. Please click here to log in to your Acme Corp account to retrieve your Download Key and see the status of your subscription."). However, I don't consider this an MVP goal, unlike the other improvements already made. If this PR is accepted I can make a proposal for further improvements and put them up for discussion -- I am not the only extensions developer out there with paid extensions, after all :)

avatar brianteeman
brianteeman - comment - 23 Oct 2019

I was just thinking aloud and if I had come up with a meaningful alternative text I would have proposed it - but like you I couldnt ;)

avatar brianteeman
brianteeman - comment - 24 Oct 2019

Quickicon 1

The text says "Download Keys entered correctly"

This is confusing to me as I took it to mean "Download Keys are valid" when it actually means "all extensions that support a download key have something entered in the field"

I can't really think of any suitable text. I was wondering if as you don't display the quickicon at all if there are no extensions requiring it that we don't display the quickicon at all if all DK have been entered?

Quickicon 2

Some free extensions are identical to the paid for extension except for the presence of a DK and when you subscribe to the extension then they give you the DK to enter on your site. This would mean that any site using a free extension like this would have a big red warning on the home dashboard etc

avatar nikosdion
nikosdion - comment - 24 Oct 2019

@brianteeman Regarding item 1: Note that I am using the Download Key feature which was created as part of GSOC. I am not changing how it works. The way it determines a "valid" key is when the extra_query in the update site record kinda-sorta follows the prefix/suffix pattern specified in the extension manifest's dlid attribute. Therefore a non-empty extra_query in the update site record which kinda conforms to that format results in a "valid" Download Key. The final verification of the Download Key is of course determined when downloading the update.

I am against adding any kind of verifying the Download Key in any other way because it'd need a request to the developer's site. To cut a long story short, the amount of traffic generated would require a massive increase in software delivery infrastructure cost (we're talking about two orders of magnitude) which would mean that developers would simply not use Joomla's integrated updates because it's financially unviable.

If we were to solve this, it would make sense to expect developers to reply with HTTP 402 Payment Required when the Download Key is invalid and have Joomla report this differently, e.g. showing a message to the user that the Download Key for XYZ extension is invalid and they probably need to go to the developer's site to see what's going on. There are ways to improve Joomla's response after a failed download without screwing over developers but I think it's outside the scope of this PR. I could work on that separately but I'm not sure if the Joomla Project is interested in it. Not having a roadmap doesn't help here.

Regarding item 2: The free version of the extension does not have the dlid attribute in the manifest, therefore it does not participate in the Download Key feature. As a result it's not participating in the notification of the quick icon plugin. I can't rule out the possibility that a developer will stupidly add a dlid attribute in a free extension but that's a problem between the chair and keyboard at the developer's side, not something to address in the Joomla core -- I am saying this as the idiot who actually did that stupid thing while trying to figure out how to use the new Download Key feature.

avatar brianteeman
brianteeman - comment - 24 Oct 2019

I never said anything about doing a validation (I agree that is wrong) I was pointing out that the text used implies it is validated and therefore in the absence of better text I was suggesting

I was wondering if as you don't display the quickicon at all if there are no extensions requiring it that we don't display the quickicon at all if all DK have been entered?
ie we only show the icon when there are missing download keys.

As for the second - we both know that if allowed developers will always do something daft. I guess they will soon realise.

avatar nikosdion
nikosdion - comment - 25 Oct 2019

I know what you wrote, I just followed the idea to its logical conclusion. I know that if a bad idea is not said out loud with an explanation of just how bad it is and why someone will eventually think about it, believe it's a good idea and may go even as far as implement it in the core :)

To get back to this issue, I don't think that making the icon disappear if all is entered correctly is a good idea. The icon can serve as an indicator of whether paid extensions are installed. Maybe we can change the language string but I'm quite literally at a loss for words. How can we succinctly convey that the user has entered something for the Download Keys which may or may not be correct but can only be verified when updates are being downloaded? If anyone can come up with a better wording, please, by all means!

avatar richard67 richard67 - test_item - 25 Oct 2019 - Tested successfully
avatar richard67
richard67 - comment - 25 Oct 2019

I have tested this item successfully on df7bb23


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/26744.

avatar Quy Quy - test_item - 25 Oct 2019 - Tested successfully
avatar Quy
Quy - comment - 25 Oct 2019

I have tested this item successfully on df7bb23


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/26744.

avatar Quy Quy - change - 25 Oct 2019
Status Pending Ready to Commit
avatar Quy
Quy - comment - 25 Oct 2019

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/26744.

avatar wilsonge wilsonge - close - 25 Oct 2019
avatar wilsonge wilsonge - merge - 25 Oct 2019
avatar wilsonge wilsonge - change - 25 Oct 2019
Status Ready to Commit Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2019-10-25 22:29:47
Closed_By wilsonge
Labels Added: ?
avatar wilsonge
wilsonge - comment - 25 Oct 2019

Thank you very much @nikosdion and @crystalenka

avatar Quy
Quy - comment - 3 Jun 2021

@nikosdion May I have permission to link to the Null file for a PR with Update Sites? Thanks in advance.

avatar nikosdion
nikosdion - comment - 3 Jun 2021

@Quy Yes, of course :)

Add a Comment

Login with GitHub to post a comment