[#43772] - [5.1.2] TUF Debug Message: Syntax error
- Closed
- 11 Jul 2024
- Medium
- Build: 4.4-dev
- # 43772
Steps to reproduce the issue
Since the update to 5.1.2 the system checks for joomla update and extension updates are all failing
Console says GET https://xxxxxxxxxxxxxxxx.com/administrator/index.php?option=com_joomlaupdate 500 (Internal Server Error)
The same is shown for checking extensions but this time the url is obviously
https:/xxxxxxxxxxxxxxx.com/administrator/index.php?option=com_installer&task=update.find&137364379be5489eaa394cbd7c689007=1
Debug stack
Line | Function | Location
--- | --- | ---
1 | () | JROOT/libraries/vendor/php-tuf/php-tuf/src/Metadata/MetadataBase.php:90
2 | json_decode() | JROOT/libraries/vendor/php-tuf/php-tuf/src/Metadata/MetadataBase.php:90
3 | Tuf\Metadata\MetadataBase::createFromJson() | JROOT/libraries/vendor/php-tuf/php-tuf/src/Client/Repository.php:67
4 | Tuf\Client\Repository->getTimestamp() | JROOT/libraries/vendor/php-tuf/php-tuf/src/Client/Updater.php:181
5 | Tuf\Client\Updater->updateTimestamp() | JROOT/libraries/vendor/php-tuf/php-tuf/src/Client/Updater.php:148
6 | Tuf\Client\Updater->refresh() | JROOT/libraries/src/TUF/TufFetcher.php:118
7 | Joomla\CMS\TUF\TufFetcher->getValidUpdate() | JROOT/libraries/src/Updater/Adapter/TufAdapter.php:84
8 | Joomla\CMS\Updater\Adapter\TufAdapter->getUpdateTargets() | JROOT/libraries/src/Updater/Adapter/TufAdapter.php:50
9 | Joomla\CMS\Updater\Adapter\TufAdapter->findUpdate() | JROOT/libraries/src/Updater/Updater.php:263
10 | Joomla\CMS\Updater\Updater->getUpdateObjectsForSite() | JROOT/libraries/src/Updater/Updater.php:164
11 | Joomla\CMS\Updater\Updater->findUpdates() | JROOT/administrator/components/com_installer/src/Model/UpdateModel.php:275
12 | Joomla\Component\Installer\Administrator\Model\UpdateModel->findUpdates() | JROOT/administrator/components/com_installer/src/Controller/UpdateController.php:113
13 | Joomla\Component\Installer\Administrator\Controller\UpdateController->find() | JROOT/libraries/src/MVC/Controller/BaseController.php:730
14 | Joomla\CMS\MVC\Controller\BaseController->execute() | JROOT/libraries/src/Dispatcher/ComponentDispatcher.php:143
15 | Joomla\CMS\Dispatcher\ComponentDispatcher->dispatch() | JROOT/libraries/src/Component/ComponentHelper.php:361
16 | Joomla\CMS\Component\ComponentHelper::renderComponent() | JROOT/libraries/src/Application/AdministratorApplication.php:150
17 | Joomla\CMS\Application\AdministratorApplication->dispatch() | JROOT/libraries/src/Application/AdministratorApplication.php:195
18 | Joomla\CMS\Application\AdministratorApplication->doExecute() | JROOT/libraries/src/Application/CMSApplication.php:306
19 | Joomla\CMS\Application\CMSApplication->execute() | JROOT/administrator/includes/app.php:58
20 | require_once() | JROOT/administrator/index.php:32
joomla-cms-bot
- | Labels |
Added:
No Code Attached Yet
|
||
Reproduced the error on two sites, now is the error gone
it's ok now
PLEASE leave this open for a while so that other people will see this
ok, in this case please close it when you think it should be closed.
@HLeithner also what can be done to prevent this (whatever it was) re-occurring. It should not rely on your availability alone to resolve this if it happens again
it was once again broken json file cached by cloudflare (actually the cache was broken noch the file). So the cache clear can be done by users with cloudflare access. on the other hand I'm planing to auto clear the cache after a new file gets uploaded by the tuf cronjob. I think it's a race condition between a 0 byte file downloaded by cloudflare while uploading by drone.
@HLeithner What do you mean by "broken json file cached by cloudflare" I am not using the Cloudflare CDN, but my own stuff, and I got the same problem... please see here https://forum.joomla.org/viewtopic.php?f=839&t=1010465
Joomla uses cloudflare as CDN for our update server, it caches the tuf json files (for j5.1+) and the xml files (for < 5.1). This files provides the update information for joomla. Everyday the timestamp.json is regenerated with an uptodate time, we upload the file from the drone CI to our webserver, this tooks a couple of ms. In case cloudflare (or one of the cloudflare mirrors) downloads the file at exact the time the file is created it's possible that it gets a 0-length file. We use rclone for upload which should prevent this error but I hadn't the time to debug it in deep. Clearing our cloudflare cache resolves this issue, that's the reason I think that this is the problem which we need to solve.
Wouldn't it make sense to add a try catch here https://github.com/joomla/joomla-cms/blob/5.1-dev/administrator/components/com_installer/src/Controller/UpdateController.php#L113 to avoid issues such as this one?
Joomla uses cloudflare as CDN for our update server, it caches the tuf json files (for j5.1+) and the xml files (for < 5.1). This files provides the update information for joomla. Everyday the timestamp.json is regenerated with an uptodate time, we upload the file from the drone CI to our webserver, this tooks a couple of ms. In case cloudflare (or one of the cloudflare mirrors) downloads the file at exact the time the file is created it's possible that it gets a 0-length file. We use rclone for upload which should prevent this error but I hadn't the time to debug it in deep. Clearing our cloudflare cache resolves this issue, that's the reason I think that this is the problem which we need to solve.
Got it and thank you for the explanation. But as posted in the forum how could this relate to the other 2 cases, how comes that the overrides and privacy requests get blocked too? As you can see in the screenshot there are 3 buttons that turned red?
Anyways, good news is that it is now fine. I don't see that anymore, at least not on the websites I checked.
I have seen this on a prior update - I believe it was the one updating to 5.1.0. Have never had it prior to that one and I manage 80+ sites so you know it would have shown up somewhere if it was a previous, older issue.
Last time it seemed to only effect some sites. This time it was every site. So, it's not a new issue, newer compared to older Joomla versions. So what has changed? If it never happened in the past (I've been around since v1.5), what is being done differently now compared to then? If it's now occurred twice, what's to stop this every time we get an update?
But as posted in the forum how could this relate to the other 2 cases, how comes that the overrides and privacy requests get blocked too? As you can see in the screenshot there are 3 buttons that turned red?
without looking at the code I can't tell you the reason, possibly by chance? or some data broke the ajax request. without looking at the response I can't answer this question.
Last time it seemed to only effect some sites. This time it was every site. So, it's not a new issue, newer compared to older Joomla versions. So what has changed? If it never happened in the past (I've been around since v1.5), what is being done differently now compared to then? If it's now occurred twice, what's to stop this every time we get an update?
Since 5.1.0 we have TUF in charge for managing updates, more information at https://magazine.joomla.org/all-issues/april-2024/tamper-proof-core-updates-for-joomla-tuf-making-it-into-5-1
it's not intended that this happens with each update....
I received emails this morning and last night from 4 sites (out of 30 5.1.2 sites). When I log into the sites I'm not seeing any errors.
I went to 'update extensions', and 'update joomla'.
I have not received any emails in about 4 hours. I assume it is fixed?
thanks,
Laura
Thanks! That explains what changed. Seems the change may have provided us with "tamper-proof updates" which is great, but is failing in conjunction with outside influences (Cloudflare's cache in this case).
Strange that with the 5.1.1 update I didn't get his at all.
@rytechsites yes, sites are back to normal. As I understand Joomla's operation, it polls the update server when a site is accessed to see if there's an update. With the corrupted (for want of a better word) file, it got bad info. Garbage in + garbage out! Once that file issue had been resolved, then the problem went away on its own.
| Status | New | ⇒ | Closed |
| Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2024-07-11 19:25:43 |
| Closed_By | ⇒ | brianteeman |
Hello, why is this issue closed? I'm having this problem with many sites on 5.1.2 and can't solve it by just clearing the cache. Where can I find the corrupted cloudflare file?
Hello, why is this issue closed? I'm having this problem with many sites on 5.1.2 and can't solve it by just clearing the cache. Where can I find the corrupted cloudflare file?
Hi. You misunderstood the issue, as did I a bit though I was asking this in the forums, it is on their side that Cloudflare does the caching (the Joomla's infrastructure), not on ours. Simply wait. Though it doesn't make sense to affect, in my case, the privacy requests from users.... One thing is clear - that update thingy is tied into way too many places.
One thing is clear - that update thingy is tied into way too many places.
??
One thing is clear - that update thingy is tied into way too many places.
??
Look at the screenshot, what do the Privacy: Information Requests have to do with an external CDN? How does this make any sense?
same proble to all my sites
All my sites (14) are haveing the same problem :(
What i also see:
https://xxxxxx.com/administrator/index.php?option=com_joomlaupdate&task=getMenuBadgeData&format=json
Update: the work around for me was enable :
extension=sodium
in the /etc/php/8.3/fpm/php.ini
I found this thread when investigating why J5.1.2 is SO slow to check extensions.
It would seem this is related to the issue with the errors you have seen. I don't get those unless I click on something to go away from the "Dashboard". When I do, they all go red.
I just figured that was something with the updates. But that's been happening for the last 2 or 3 version updates - that I have to wait an inordinate amount of time for the system to check extensions and overrides.
How much time? Anywhere from a minute to 5 minutes...or until I get frustrated and just say, "Never mind!" and then go about my work. It's been months since I've seen the "norm" - 3 to 5 "seconds" for checking.
I usually only wait now if I know I haven't updated recently. If it's been more than a couple of weeks, I wait for the "checking..."
It also takes a long time after you click on any update found from the dashboard to get to the updates.
If this issue with Cloudflare is something someone found in July, I can assure you, I've been seeing it since May or June. But my "wait" is not for anything other than checking extensions. I don't have reason to use cache in my site. It's running anywhere from mid-90's to 100 on every page, so I don't worry about cache.
Thank you, Brian, for keeping this open.
I really thought this checking extensions was something that would be so widespread I wouldn't need to post anywhere because everybody was having the issue. But apparently, that's not the case.
I know it's not my connection because this only started after a J5 version update. It's not due to any plugins or extensions either. I have Joomla in several sites and they are all exactly the same when it comes to the checking extensions - very slow. Yet, the other sites have either no additional plugins, or, whatever plugins / extensions there are differ in every site.
Normally, TUF / Joomla Update shouldn't affect extension updates... that's a bit strange, also I have no delay checking for updates. Are you able to debug this in more deep? have a side I can look at it and where it is reproduceable?
Normally, TUF / Joomla Update shouldn't affect extension updates... that's a bit strange, also I have no delay checking for updates. Are you able to debug this in more deep? have a side I can look at it and where it is reproduceable?
I just recorded 5 different installations. 1 is normal and has very little delay. 1 is a bit faster than the others, but still has a bit of a delay. The other 3 are obviously delayed by no less than 30 seconds - today. I say it that way because typically, there is at least a 90 second delay.
2 sites are only staging sites. So there are only projects on which I'm working that are sub-folders to the main Joomla 5 site. In 1 of the staging sites, there are no projects - just a default J5 installation. In a new site on its own domain, one that has no plugins or extensions added (other than JCE), that one is the one that's a bit faster, but still had a delay this morning.
I am now thinking that the 3 sites I have with Virtuemart very well may be part of the problem. I do have Virtuemart installed in the staging sites also. But all of the sites with VM have had it installed for more than a year. Yet, only in the last couple of months has anything been effected.
If the slow down was when I select an update to be installed and wait for the installation, that's completely different and I understand that's on those who created the plugin / extension.
But the "checking" is where the slow down is found which is long before I can get to where I would update the extension.
If this has anything to do with the connection to a server, then there is something that has changed in how J5 is communicating with Cloudflare because it was only in the last couple of months that this started. And it wasn't like 1 site had slowed, then I noticed in the others. It was every one of the sites that has the delay started at the same time.
It's simple to reproduce, if it's going to happen. Simply log in to admin....and wait for the extensions to be checked. I've had it sit for as long as 5 minutes before it finally resolved to the updates window. I've had it take so long that I just went about my business instead of waiting longer than the couple of minutes I did wait.
Today, the longest I waited while recording today was just "under" 1 minute. I'd say that's good except I believe the decrease in delay is directly related to the "time of day" on the Net. As we all know, depending on the time of day, connections slow down. Since I checked this morning before 08:00 PDT, the Net has little traffic.
When I was checking yesterday, I waited more than a minute for each of the sites to resolve the extension checks.
Before you blow this off as "it's his connection", you have to ask, why would it suddenly start just 2 or 3 months ago when I've been developing "from" the same area for 25 years, and using Joomla since it was Mambo? In the last year I've had no changes to my Internet connection.
I could understand if it was 1 site. But 4? And each of them have the same problem which started at the same time?
All of my sites are hosted with A2Hosting.com in my reseller account. None of the sites use cache.
If you want access to one of the sites, let me know and I'll send the credentials to you. I'm going to check later today to see if the delay is worse because the fact that they all "only" took about 40 seconds to resolve this morning is the fastest I've seen them for the last couple of months and I want to see if time of day is a factor.
I tried to replicated it on my own servers and can see a delay but nothing to long 3-4 seconds.
could you please disable all your updates site at /administrator/index.php?option=com_installer&view=updatesites and try each site separately ?
maybe one of the has an issue
I tried to replicated it on my own servers and can see a delay but nothing to long 3-4 seconds.
could you please disable all your updates site at /administrator/index.php?option=com_installer&view=updatesites and try each site separately ? maybe one of the has an issue
You may be right. Even in the staging site where there are only a couple of additions, I disabled everything and voila - extension check was immediate.
It's going to take some time, but I'll go through them 1-by-1 to enable, check, and repeat until I can see if there is 1 or more that are causing the slow down. I'll report back later with my findings.
I have one site that would take a very long time. The problem was with one of the extensions where the developer has gone and the domain has been taken over by a domainreseller. It takes a very long time before joomla realises that it is not getting an answer to the request for update information. As soon as I disable the update site for that extension everything is fine.
In short your problem is NOT related to the TUF issue this topic was opened for but is definitely because of one of your extensions.
I have one site that would take a very long time. The problem was with one of the extensions where the developer has gone and the domain has been taken over by a domainreseller. It takes a very long time before joomla realises that it is not getting an answer to the request for update information. As soon as I disable the update site for that extension everything is fine.
In short your problem is NOT related to the TUF issue this topic was opened for but is definitely because of one of your extensions.
Thanks, Brian. I've been trying to address my investigation with that in mind. I agree that it's most likely one of the extensions. Sadly, I'm thinking it's something with Virtuemart.
In order to get true results, I have to disable the extension updates, then leave the site for a while, return and check. Otherwise, once the extensions have been checked, I can be doing other things in the site, return to check updates and the checking is done in less than 5 seconds - the norm.
It's only when I initially log in that it takes forever. So I'll continue to drill down through them all (VM has 2 pages of them) and determine the culprit.
Thanks again.
No - all you have to do is go to administrator/index.php?option=com_installer&view=update and click on check for updates
I've done both methods. Wait at the dashboard, or simply click on "Extensions are up to date" (or "Checking extensions..." if that's the case). Both had me waiting a very long time before I disabled everything then re-enabled them all.
After disabling all of the extensions the enabling them 1-by-1, which I did in all of the sites, seems to have had a positive effect. No matter how I've gone to check now, I get all green within 5 seconds in the sites, and when I click on Extensions are up to date, that's also fast.
I was going to "Rebuild" all of them, but it seems to have rectified itself. I'm going to keep trying things today and see if it's completely resolved, or just a temporary resolve.
I guess there was an "air bubble" that needed to be burped... But I'm going to continue to check today and tomorrow to see that it's completely resolved. Thanks.
Can you please try again.