I have spotted a few issues with the way the Joomla extensions update works. The Joomla project is already aware of these issues. I had privately discussed them in person with @wilsonge several years ago and emailed @zero-24 about them several months ago. No action has been taken. Just like when I had reported the same issues to your predecessors between 2013 and 2016. No action taken whatsoever.

Since the JED requires developers to use this subpar updater to be listed in the JED, the Joomla project needs to either fix these issues or allow developers to implement THEIR OWN updater code. It's not reasonable demanding that 3PDs use your updater and bear the temporal, financial and repetitional cost of its inadequacies.

Seeing how certain core contributors abused the discretion expressed in my previous disclosures of these issues in private to make unsubstantiated public claims I feel the urge to report them publicly. You can no longer pretend you have never heard of these issues I have reported to the Joomla project the last 8 years. You can no longer pretend that the JED rule makes perfect sense and somehow serves the 3PDs best interests when all it does is hinder us in delivering updates reliably to our users while bearing the costs of a decision imposed upon us by OpenSourceMatters Inc.

You cannot update the extra_query after an update has been discovered

Joomla is using a weird update process. The update site has the extra_query column where the developer (or the user, as of Joomla 4) can enter an authentication query to the server delivering the update. Joomla retrieves the update information, populates the #__update table and copies over the extra_query column. I know how this part works because I contributed it for Joomla 3.2.

Before you start saying things like that I am reporting a bug I created please be advised that the fact it is implemented the way it is was NOT my choice. It was something discussed with Mark Dexter who explicitly framed my proposed contribution as something that should not change the way Joomla's extensions updates work. A few months after this contribution I had written a white paper on creating a better extensions updater that the Joomla project refused to consider. This was 8 years ago.

In the original version of the code updating the extra_query column in #__update_sites would simply require to Clear Cache on the Update page and then use Fetch again to retrieve the new updates, copying over the updated extra_query column. Further to that, Joomla would display the server response which would be used by the developer to communicate that the user needs to enter the correct download key and refresh the updates. The latter was removed for security reasons but it's largely irrelevant since Joomla 4 does notify the user when an update key is missing.

The problem is that it's no longer adequate to Clear Cache and fetch the updates again. The reason is that the queries are cached. There is a very delicate and frustrating 14 step dance you have to do to ACTUALLY be able to use a new download key in Joomla 4 when updates have already been found.

This has been a recurring issue which is costing us a great deal of support time and client goodwill on each and every update we publish. Every time we publish an update we know that we have to waste two days telling people to follow the 14 to 18 step process. This has happened more frequently ever since the rebuild button was added to the Update Sites management page, a feature which removes the extra_query column without the user ever knowing about it.

This issue is largely fixed in Joomla 4 EXCEPT for the part concerning the extra_query not being used unless you also clear the administrator cache.

Proposed solution: clear the query cache when you clear the update cache.

Files are not copied on update

This is a problem we have experienced since 2013, when we first started using the Joomla extensions updater instead of our own updater.

When installing an update to an extension Joomla will sometimes not replace all of the files included in the extension package.

This does not happen every time. It happens to about 3% to 5% of updates or at least that's the number we can infer based on support tickets. It might be substantially higher, just people not having realised because the file / folder not copied was either unchanged between versions, was about a feature they are not using or didn't cause a visible issue the user would report.

Our solution was to tell users to install the extension manually, twice in a row, without uninstalling it before or in between as you can see in this ticket from 2013. For some reason, the second installation attempt seemed to work. The only difference we could discern is that the version number was the same, therefore the cached XML manifest was identical between the version supposedly installed on the site and version being installed as an update.

In an effort to find a solution to this issue which doesn't require our clients to do a double installation — a completely unrealistic approach when you have dozens to hundreds of sites, especially since the advent of site management services — we have tried the following solutions over the years:

* Listing each and every file in the XML manifest instead of the top level files and folders. It had no effect and was quickly abandoned.

* Shipping a list of files with their file sizes and CRC32 sums and check the consistency of the files when the client tried to use the extension. This was problematic if an FTP transfer in ASCII mode was involved at any time. Furthermore, the list of files and checksums was also subject to Joomla's bug of not copying files, rendering the whole exercise a moot point. It was quickly abandoned.

* Rig our manifest script's `postflight()` method to go through the files shipped in the extension package (using the `$parent` adapter passed to it by Joomla) and comparing EACH AND EVERY ONE of the files' sizes. We didn't use a checksum because on larger extensions it can cause a timeout error in PHP or even the web server itself. If the file seems to be different we copy it manually. **THIS WORKED**. This is the only thing that made 7 years of supports requests about this issue come to an abrupt halt. FEF was the only package we ship that lacked this code, hence it bumping into that problem two days ago.

Observing the results it looks like Joomla definitely has an issue here. I suspect it has to do with how it handles the XML manifest file before using it to determine which files and folders to copy on update.

This issue did happen with all of the updates we published on March 2nd because one package (FEF) did not have the workaround code.

It is, however, unreasonable to expect 3PDs to essentially implement an updater inside the updater to work around your updater being broken. It is equally unreasonable expecting users to use Upload and Install or Install from URL instead of the Update page to install updates, especially when there are web design companies with entire fleets of dozens to hundreds of sites.

Proposed solution: fix the installer so that it always uses the manifest information present INSIDE THE PACKAGE being installed. Alternatively, let 3PDs implement their own updater.

OPcache

Joomla's extensions updater essentially replaces parts of the site itself, including executable code (.php files).

Unlike what was the case in 2010, it's not guaranteed that writing a file to the server's storage will result in that file being loaded. Modern PHP-based sites are very likely to use OPcache to accelerate the site by caching the parsed opcode of frequently used .php files. Given the way Joomla works, when installing updates this definitely includes the files of system and installer plugins and any code they loaded during the request which initiated the update and the request before it (this can include components' and other extensions' .php files).

This is highly problematic for two reasons:

* If the post-installation code in the XML manifest script relies on the newly installed files being loaded it's very likely that they are already in OPcache and that OPcache has `opcache.revalidate_freq` set to a value higher than the time it takes for the extensions updater to download and install an update. This also applies when doing a manual update using Upload and Install or Install from URL. This means that the post-installation code is being served older versions of these files which might not be compatible with type hinting in its methods or otherwise cause fatal errors due to the disparity between the expected and the installed version. I was bitten by this exact issue with the Akeeba Backup 8.0.0 and Admin Tools 6.0.0 updates two days ago.

* After the update is complete it's possible that the server uses the old code of system plugins but loads the new code of a component or other extension these plugins are dependent on. Since the check to see if the extension they depend on is installed and activated goes through Joomla's ComponentHelper et al is uses database data. Checking files for existence or even their content is also not reliable because the files do exist, their content is what it should be but PHP is using an older version of the file which is no longer there. I was also bitten by this issue in the same versions of my software.

I thought I had solved this issue by using opcache_reset() in the beginning of my postflight() method. This is the same trick I am using in the Joomla Update code I contributed to Joomla in 2.5.1 for the exact same reason. However, the PHP documentation is wrong or, at the very least, incomplete. opcache_reset() does NOT reset the opcode cache immediately, even when it returns true. What it really does is that it WILL reset the opcache when PHP is exiting. This solves the second issue but not the first. I was bitten by this in the Akeeba Backup 8.0.1 and Admin Tools 6.0.2 updates.

The solution was to use opcache_invalidate() on each .php file being installed which does work as documented, i.e. it immediately invalidates the opcache for the given file. Akeeba Backup 8.0.2 and Admin Tools 6.0.2 prove beyond any doubt that this approach does work. This is critical for the proposed solution.

Proposed solution: since Joomla already has code to copy each and every file in the extension package it should check if the file being copied has a .php extension and invalidate the OPcache for the file's final location. This should happen conditionally, so Joomla doesn't break if OPcache is not installed. function_exists() is your friend.

Quoting this here just to prevent whitewashing this report here later on. 😘

Mr. Papenberg's response is extremely problematic because he's speaking as an official of the Joomla project on an official communication channel, making this the voice of the project. Does the Joomla project condone character assassination and slandering on its public communication channels?

At the minimum I would expect to get a public apology from the officials of OpenSourceMatter Inc (OSM) and the Joomla project and the immediate termination of Mr. Papenberg from the project as the minimum restitution of the reputational damage effected upon our company and me personally by his patently false allegations. I'll come back to that.

I would also like to see these long standing bugs I have been privately reporting for EIGHT YEARS fixed. I had not publicly disclosed them before because they make Joomla look like amateur hour. I had told you in private, you chose to systematically ignore me and now you pretend we never had these conversations. So be it, here's a public issue documenting the top failures of the Joomla extensions updater which is forced upon all 3PDs. I'd have very much preferred we kept this private and worked together to fix these issues.

And now let's get back to the patently false allegations.

For those unaware, Mr. Papenberg started by making abjectly false, public allegations on Twitter that I blamed Joomla about all problems we experienced with our latest updates. Screenshot below (I don't trust him to not cowardly delete his tweet):

The ACTUAL report I have written and to which he alludes can be found at https://www.akeeba.com/news/1744-advice-for-install-probs-with-march-2021-software-releases.html

As it's painfully obvious, I am talking about 5 issues which occur in a number of sites (NOT all sites), only ONE of which is directly attributed to Joomla, namely the installer not copying files on update, a bug I have been reporting to ten or so different people over the past 8 years and which the project STILL pretends doesn't happen.

The other four issues were never attributed to Joomla. Joomla's involvement in them is merely coincidental, insofar updates to our Joomla extensions are executed by Joomla itself, this being an objective truth. Could they have been avoided had the Joomla project not forced us to use its subpar, buggy extensions installer? Yes, I believe so. However, it's not a point I made in the article.

The technical explanation in the article of how Joomla works with regards to extensions updates is objective and verifiable by reading Joomla's code. The reason it was made was to provide the background knowledge to those who cannot read Joomla's code. Otherwise one could reasonably wonder why doesn't the pre-installation code in the XML manifest script address all of them. It's an objective fact that any 3PD can only do whatever the execution environment allows them to do, in this case it being Joomla and PHP. I am not blaming Joomla or PHP — even though I am equally surprised and miffed that the PHP documentation on opcache_reset is wrong. I am merely saying that we didn't bump onto these objective limitations during testing AND what we did to make sure that next time we will.

Since Mr. Papenberg makes it sounds like a complete disaster, it was far from it. The failure rate of the update was very low, about 1%. Still, 1% of a few hundreds of thousands of sites is a big number. Compare that to the accidental or deliberate but still not communicated before release backwards incompatible changes effected by Joomla over the years which broke sites. Also compare our response to addressing a massive, extremely complicated update breaking for a stark minority of users to Joomla's response for messing up a simple one liner change making a whole class of folder names impossible to use in Joomla 3.9.25. Both issues happened at exactly the same time. Within the exact same timeframe we worked to provide a simple solution before making a public post about the resolution steps and offering a detailed post-mortem. Joomla did nothing except publish a page two days later saying "oops, sorry, we broke this and it will remain broken, here's how broken it is". I think that our response was far superior to that of the Joomla project. We took responsibility and we fixed it instead of saying "Joomla and PHP do things we could have never expected, oops, it will remain broken, figure it out yourselves", i.e. we didn't pull a Joomla...

As for the last bit in the article about why beta releases were not made, it's an objective fact corroborated by the email exchange I had with Mr. Tobias Zulauf (@zero24) in late January. I had already told him that the JSST's FALSE AND WRONG "security" issue puts us in an impossible position, messing up with our release schedule. I had also fully explained why the code they marked as "potentially insecure" is in fact an implementation of RFC 4086 and linked him to the article of the very person who provided the Joomla project with guidance in writing that code (the code the Joomla project falsely attributed to our company). If I was a suspicious person I'd think that this is an orchestrated smear campaign. However, knowing Joomla, I am not going to attribute to malice what can be readily explained by incompetence.

As for the wild allegations made in Mr. Papenberg's reply to this issue that I might "whitewash" my issue, I reject them with disgust and horror! I have NEVER done so and claiming otherwise is slander, pure and simple.

I have closed issues and PRs when I've hit a brick wall. I have deleted the branches on my own repositories for PRs I closed. I have NEVER, EVER edited my issues to change their meaning! The history of my issues and PRs is publicly available on GitHub and so is the edit history. You can also find my history of contributions and issue reports before GitHub on the JoomlaCode tracker archive on the Joomla! Issues site.

Maybe you will find it interesting to see that the Joomla project has been systematically abusing me since 2009, when I first attempted to contribute back to it. Look for the issue where I explain that VARCHAR(2000) and TEXT are functionally identical in MySQL with regard to on disk table space which one Mr. Andrew Eddie, Joomla co-founder and then lead developer, refuted and told me I don't know how to write code — even after linking him to the official MySQL documentation proving me right. Or how the same Mr. Andrew Eddie came to his senses a year or so later and replaced the artificially limiting VARCHAR(2000) with MEDIUMTEXT, effectively proving that I was right all along, without bothering to apologise for his past comments.

All my issue reports come after researching the subject at hand. I have filed five or less issues where I had not had the time to research in advance. In those which were invalid I apologised and closed them myself. I am not unreasonable. I am an Engineer. If X is broken because of Y I will tell the person or persons responsible so they can fix it, typically proposing a way to fix it. However, if the Joomla project starts with the understanding that if it's outside the echo chamber it's invalid and you need to belittle and attack the person reporting an objective problem then it's the Joomla project's problem for being toxic, not mine or anyone else's.

Such has been my experience from this toxic project over the years. I have contributed my code, my time and my own damn money in good faith, propping up the project because I believe in its community, not the people who run it. Whenever the project found the going getting tough they tapped me to help them be it a Joomla Update component that actually works on shared hosting, Two Factor Authentication, WebAuthn, fixing zero day security issue (even after not wanting me in the JSST...) or a RAD framework to name but a few. On the latter, Joomla actively caused problems with our company's products by reneging on its promise to allow us to continue providing updates to it, shutting us out of our own framework which we had to fork (we had fork our own code!) and which led to a chronic confusion among users about the different "flavours" they see installed, see https://www.akeeba.com/news/1558-info-about-fof-and-f0f.html

Because of this chronic toxicity I made it a point to no longer contribute anything back to the Joomla project starting January 2021. I have given up on Joomla. Is that not enough for you people?

This issue is not an attempt to contribute. I will NOT fix Joomla's code even though I know how to and had proposed time and again to do that over the past 8 years. I am just publicly disclosing all the issues with the Joomla extensions updater so you can FINALLY stop pretending they don't exist. Your duplicity has gotten beyond annoying.

So, fix your damn code or let us 3PDs provide our own. Forcing to use your own solution, pretending that it has no issues, fixing nothing we report and having us bear the burden of doing end user support for this broken mess is unethical and coercive. I'm leaving it at this.

If you need further information you know how to contact me. I am not going to participate in this charade where disclosing issues about your code is met with slander and character assassination. You are welcome to drown in your own toxic, septic tank. I'm outta here.

BTW @Hackwar github has a change tracking implemented, your post is neither helpful

The thread in question on twitter is here: https://twitter.com/Hackwar/status/1367769460208197632

Screenshot below (I don't trust him to not cowardly delete his tweet):

Please know that the akeebabackup handle itself deleted 6 of their own tweets out of the above linked thread. I couldn't remember that I ever deleted a tweet or a response of mine on github afterwards. The above quote of the original issue text by me has been done, because in contrast both the akeebabackup and sledge812 handle on twitter and nikosdion on github have a LONG history of deleting their earlier remarks.

@blueforce I know, however it requires awareness of the fact that a post was edited and that you have to look at the history. If you use these tactics as a weapon in a propaganda stint like in this case, then this feature doesn't really help.

And of course what I do on my private Twitter account is totally up to me. The Joomla project is in no way involved in that and your allegations have no standing. However if you feel I have slandered your reputation, please sue me, so that a court can decide on this. Otherwise if you want me to lose my position in the Joomla project, that is rather easy. Organise a new lead for the ATT and I'll organise a new election right now.

This conversation is getting out of control.
The issue is not imho to attack each other, going back as far as 2009 and targeting Andrew or tweeting without control or talking about sueing,

The real issue is: "Is there a problem or not with the update installer?"
If there is, everybody has to join forces to solve it.

As Joomla Grand Pa, I lock this issue now.

As there is apparently a problem I created #32597

Discussions and proposals to solve are welcome there.