@PhilETaylor Not sure if it's an issue, but theoretically, as the update sites table doesn't have any unique index on the location column, it can be that there are several update sites with the same location but different extra queries. Practically I think it's unlikely because that wouldn't make much sense.

That would make no sense like you say. There is a comment somewhere about multiple update sites and not allowing it.. let me see if I can find that again...

This code stops the same location being inserted into the db table twice.

@PhilETaylor Good that we checked.

Next question is where do I find an extension which needs an update query. Do you have some examples? I don't use such on my private site.

They are mainly paid for updates, inc Akeeba and Admin Tools ... email me direct for test copy.

You dont need a valid download id for them, as you can die after echoing the URL that will be used, and that url should confirm that it will use the extra

Die on this blank line with echo $url; die;

@PhilETaylor Would a hello world component plus an update site which are extended as described here https://joomdonation.com/forum/os-property/24123-extra-query-not-working.html plus some redirect rules in .htaccess for the extra query URLs to make sure the an URL from version x is not used with the next media ID query parameter of version y and vice versa, i.e. return a 404 if it doesn't match, do the job? I could provide the update site with the redirect rules, but with the component I'd need help maybe.

That is overkill for the purposes of testing. Nothing has changed with the actual remote checking of an XML stream or the application of an update.

The only things changed are how Joomla internally handles rebuilding the update sites table and how Joomla internally handles the application of an extra_query on an already cached update.

You can use ANY old extension that has an update available, and manually edit the db tables. I'll try to write up some instructions to test that route after lunch.

I'll try to write up some instructions to test that route after lunch.

Bon appetite!

Testing instructions have been added to the opening post of this issue

All four bugs fixed have instructions.

Thanks for testing, I know there is a lot of set up and time needed to test, but honestly this fixes a bug that has plagued Joomla for years.

Will test later today or tomorrow.

I have spent sometime to review the code and have few questions (hopefully it is not stupid question because I don't understand the code/logic of the whole update process):

• Should we always use extra_query from #__update_sites table instead of extra_query from #__updates table? That should help prevent using an outdated extra_query (when updates are found and cached before extra_query is updated by user - the case extra_query needs to be updated)
• Do you have a better way to pass extra_query to the plugin? For example, pass the parameter in the plugin trigger here https://github.com/joomla/joomla-cms/blob/staging/administrator/components/com_installer/models/updatesites.php#L352 ? Passing it as a property of Installer class is a bit ugly.

I also made PR PhilETaylor#1 to your branch with suggestion to improve code to save the system from have to run for each loop. Please review to see if it looks OK for you and If yes, merge it.

Also, in the issue #32592, the author suggested this solution:

Proposed solution: clear the query cache when you clear the update cache

Do you know what query cache is he talking about? Would be nice if we can figure out solve all the reported issues with extra_query on this PR.

Do you know what query cache is he talking about? Good be nice if we can figure out solve all the reported issues with extra_query on this PR.

@joomdonation I'm not sure, but maybe Nicholas means the MySQL query cache, which is something used by MySQL versions lower than 8.0.2 but has been removed with 8.0.2, while MariaDB still has that query cache. See e.g. this PR here: #32028 .

@richard67 Maybe not because he mentioned about clear Joomla cache in his instructions for his customers.

@richard67 Maybe not because he mentioned about clear Joomla cache in his instructions for his customers.

That's why I wrote I'm not sure.

@PhilETaylor I've found a few code standards issues, like not using names quoting in that DB query which has been modified with the last commit to use "TRIM", or with the function parameter $extra_query using underscore instead of camel case. If you want, fix it, or let me know when I should suggest changes or make a PR to your branch to make it easier for you. Personally, I can also live with it if we fix it with a later PR, if you say "don't bother me with that now".

Sorry, I just hit the merge button, I have other work overdue for the weekend I need to concentrate on after spending the last 3 days in open source land. the code passed all the automated checks.

I'll be happy to merge any PR's you make against the branch.

I'll be happy to merge any PR's you make against the branch.

That's ok for me.

Merged

This one would be a good one to test so that it can make it for Joomla 3.9.26.... if maybe.

Stay tuned, I will test tomorrow or on Friday or weekend . Same for the other PRs.

Thanks - I dont know when 3.9.26 is being cut... but I was kind hoping it was this week ready for release next week :(

I don’t know when 3.9.26 will be prepared, but I’m clarifying. I am too tired now to test anything today. As said, I hope I can use our Easter weekend.

@PhilETaylor I have reviewed the code and I believe that it would address the issue. However, I have few concerns which I mentioned at #32862 (comment) . Do you want to take a look at it?

Should we always use extra_query from #__update_sites table instead of extra_query from #__updates table?

Maybe, but the purpose of this PR was to be light touch, and not to change too much, not to introduce any huge issues at the end of the Joomla 3 series with the hope that this would quickly be tested and merged to Joomla 3.9.26 (the second version that was never meant to be released...) as most of these issues are already resolved in Joomla 4, but it would be nice to fix for those staying with Joomla 3 for the next decade.

Do you have a better way to pass extra_query to the plugin? For example, pass the parameter in the plugin trigger

Could do, I have no preference either way, but again was trying not to make any method call changes (but I guess a new array key can't hurt right...)

@PhilETaylor Leave it as how it is for now. I will test it ASAP

Received. Thanks !

After looking more at how update works, I believe we should always use extra_query from #__update_sites table instead using extra_query from #__updates table. That will make it easier for users to perform an update in case they has to update the Download ID (in case he enters a wrong Download ID before)

Currently, if user has to update Download ID, he will have to:

• Update the Download ID in the extension
• Go back to update screen:

• Clear Cache to remove the found updates
• Click on Find Updates button to find updates again
• And then perform the update for the extension he wants

If we use extra_query directly from #___update_sites table, users will just have to:

• Update the Download ID in the extension
• Go back to update screen, perform updating for the extension they want

That simplify the process and also make more sense. Users won't know that they have to clear cache, then find updates again after updating the Download ID in the extension.

With that said, I made PR PhilETaylor#4 to your branch. Please think about it and if you agree, merge it so that we can move forward.

Will test and merge when away - currently 4:46am :-) and I'm asleep

Will test and merge when away - currently 4:46am :-) and I'm asleep

Why wake up and answer at this time :( ? Go sleep :).

Why wake up and answer at this time :( ? Go sleep :).

The joys of running a global business, you never sleep, you always have one eye and ear on incoming emails and alerts ;-)

I have merged your changes - thanks.

I have tested this item ✅ successfully on 4a52cfd

Real test and carefully code review, too.

I have tested this item ✅ successfully on 4a52cfd

RTC

Thanks again to those who took the time to understand this issue and tested the PR, I appreciate it. Thank you.

Thanks again to those who took the time to understand this issue and tested the PR, I appreciate it. Thank you.

You are not alone :D .

Thanks

Also, in the issue #32592, the author suggested this solution:

Proposed solution: clear the query cache when you clear the update cache

Do you know what query cache is he talking about? Would be nice if we can figure out solve all the reported issues with extra_query on this PR.

@joomdonation @PhilETaylor Meanwhile I think he meant the Joomla cache for particular components like system, com_installer and a few others, when update sites have been rebuilt. Possibly we should check that and make a follow up PR.

@richard67 Actually, he is talking about caching the updates found (Joomla! has that set to 6 hours by default). By using extra_query directly from #__update_sites instead of #__updates table as we are doing now, I think the issue with extra_query should be fully sorted by now. See the attached screenshot to see the caching I am talking about here

s/__DEPLOY_VERSION/__DEPLOY_VERSION__ #33038

@joomdonation @PhilETaylor Meanwhile I think he meant the Joomla cache for particular components like system, com_installer and a few others, when update sites have been rebuilt. Possibly we should check that and make a follow up PR.

" regarding the query cache, Joomla needs to delete the _system cache which caches Models' queries to the database. Otherwise changing the extra_query in the #__update_sites has no effect. The next time Joomla tries to fetch updates it will call the same model, use the same _system cache and end up using the old extra_query"

#33040