question regarding the security part of your post. Would it not be beneficial to check the hash of the zip against the published hashes at the beginning of the process? (note I have yet to read the code so maybe it already does - don't shoot me)

@nikosdion could you update your instructions please to say that you cannot run build.php on a windows system

This script is designed to be run in CLI on Linux, Mac OS X and WSL

Will I get shot asking for this to be covered by 100% unit testing before being merged?

@brianteeman Checking the hash is done before running the extraction already. It's meant to ensure that whatever we downloaded is what the update server is telling us we should have downloaded to perform the update. Checking that again before extraction, right after we already checked that, would be a waste of time.

Regarding build.php, dunno, I found it this way ? I thought that everyone who'd try this would know that build.php requires a *NIX system. FWIW you can run it on Windows, under WSL. Whether this is a sane thing to do is a different story. An alternative to that is take an existing Joomla 4 update package and replace the administrator/components/com_joomlaupdate and administrator/components/com_admin folders with those from this PR.

@PhilETaylor We've had the experience of writing unit tests for this in the Akeeba Backup repository and one thing we quickly realised is that it's far more complicated than your typical tests and completely nonsensical in context. The code doesn't lend itself to unit testing as it goes and works directly with files. Having it work against data in a memory buffer or stream makes it substantially slower and uses much more memory which makes it fail on cheap shared hosts i.e. exactly where we need it to work (fast hosts could just as well use PHP's ZipArchive and be done with it). One way around that is faking a filesystem in memory e.g. with vfsStream. That's exactly what we had done. Then you need to create doctored ZIP files to simulate error conditions. This requires a hex editor, a copy of the ZIP specification (APPNOTE.TXT), a lot of experience and plenty of time. I actually did some of that and it was... doubleplusunfun.

Then you realise that the one thing you cannot test is the timer code. You can mock it and you can have it return a fake out of time message conditionally... but so what? Have you actually really tested that this will prevent a timeout on most servers? No. That requires something that cannot be tested: experience debugging on these servers and a good sense of how they work both as a complete system and as individual software components. Unfortunately, there's only one of me in Joomla and I've found it impossible to train someone else in this dark art. Davide is working with me for eight years, there are still a few cases each week I have to provide input based on my experience with the dark arts of hosting environments. So, yeah, I could spend the next month writing Unit Tests but they wouldn't really be testing anything useful.

If someone wants to write integration tests that would be FAR more useful. If you have ideas how to do that you're more than welcome to contribute that! I can tell you how I did that for Akeeba Solo which has an updater. I'm creating a new installation of the previous version, I update the updater files (since that's what I'm trying to test) and apply an update created out of the repository's files. At the end of the integration test you can also test that all files have been extracted with the correct name, size and checksum.

Regarding the “Invalid login” it's INTENTIONAL. Before 2014 you would get a different message depending on the actual problem the code ran into. This made a Padding Oracle attack much easier. SO now we're not using encryption why don't I just change these messages? Glad you asked! I am preventing information disclosure which would help an attacker. If you get a different message depending on whether update.php is created, if it's there but has an empty password, if you sent no password or if the password you provided doesn't match you can tell if there's a leftover update.php from a failed update and proceed to brute force it. Showing a generic message makes it much harder for the attacker to know what's going on, meaning they can never be sure if they are brute forcing a password or they are wasting their time.

For code comments I'll reply inline.

First of all: thanks a lot @nikosdion for that PR! I've worked with the old restore.php because of my involvement in a Joomla management SAAS and the new file is clearly an update in terms of readability and maintainability!

Just one remark:

There is a catch, though. The users who have followed the instructions of the Joomla Security Wiki page on .htaccess files, have used my Master .htaccess (which is used in the Joomla wiki) or are using Admin Tools Professional's .htaccess Maker (or something similar) will need to update their .htaccess files before running Joomla Update AFTER installing whichever Joomla version includes this patch. Same goes for NginX configuration and IIS web.config files.

I want to highlight that this issue will affect a considerable number of sites. In our own SAAS-context, roughly 15% of the sites are affected and therefore will require a manual adjustment by the site owner.
So, to help those owners, I would suggest to at least add a specific error handling and some documentation around it. Somethink like a specific check for a 403 response in the JS file, triggering a message for the users, pointing them to a proper docs page explaining the issue and giving instructions how to fix it.

@SniperSister Thank you for confirming my suspicion. I don't have hard numbers, I can only extrapolate from the number of unique Joomla sites that ping our stats server for any of our software and the number of unique sites that ping our site for Admin Tools. My number was about 25%, about half would be using the .htaccess Maker based on experience so we seem to agree. That's good.

My problem is that I am making a PR to Joomla, not the third party code that added the .htaccess / web.config etc change — even though the most likely third party code owner is me. I don't want to put a message that's a self-advertisement inside Joomla.

Moreover, I cannot introduce a new language string because this PR would only make it into 4.1 which might be a long way away yet.

The best thing I can do is that we need to update the Joomla documentation. If the file is blocked you will get a dialog reading “ERROR: AJAX Error”. This means that the browser received an HTTP error response. The first thing to check would be whether you can access /administrator/components/com_joomlaupdate/extract.php. If you can and get a JSON-encoded Invalid Login error the problem with the extraction was something else. If you get a 403 you need to check your .htaccess file (if you're on Apache ro Lighttpd), or your web.config file (if you are on Microsoft IIS) or your NginX configuration. If there's nothing blocking that file there you need to check any CDN configuration or talk to your host. I'm pretty sure a native English speaker can take that and create something easy for people to understand.

I could also make it so that the action after an AJAX Error message is directing the user to the documentation page for Joomla Update. No questions asked, here are the docs, read them. Better than have a message with a link “Click here to read the troubleshooting documentation” which invariably leads to half of the people taking a screenshot of it and asking what to do (click the bloody link is what you should do, dammit!). Sorry, this happens so often in support that it is borderline triggering.

What do you think?

Moreover, I cannot introduce a new language string because this PR would only make it into 4.1 which might be a long way away yet.

Not aware of that rule

@brianteeman I have been told in the past that there's a language freeze before the beta of the x.y.0 version and no language strings are allowed until the next minor version. If I am allowed to add language strings (and I need official confirmation for that) I can definitely make a MUCH MORE USER FRIENDLY error reporting. Like, having a proper modal dialog with actual troubleshooting information instead of a JS popup. I really wanted to do that but without new lang strings this can't fly :)

@nikosdion The language freeze was between the last RC until 4.0.1, not until 4.1, see #34685 ... so now after these releases all is normal as usual, no limits for language changes.

@richard67 Thanks for the tips regarding lang strings and the git issue. I will work on better error handling since I now know I can use new lang strings :)

@nikosdion Regarding the testing scenario "update a J4 with the changes of this PR applied to a later version with the changes still applied": It doesn't really need to create own packages.

You can apply the patch of this PR e.g. with git patch or with the patchtester on a clean, current 4.0-dev and then update to the patched package built by drone for this PRm using the custom update URL of the update package which can be found when expanding the ci checks at the bottom of the PR ("show all checks") and then using the "Details" link at the right hand side of the "Downoad" line.

Since the version of the patched package update has the PR number appended, there will always be found that update even if already being on the latest "*-dev" version.

After such an update, the database checker will and should show only one problem for the CMS about not matching update versions. That is expected but should always be checked after such a test because if there is an error, there will be more problems shown.

This means that the browser received an HTTP error response.

@nikosdion what I was thinking about was to actually check the response status code in the JS and add an extra case for a 403, outputting a more targeted error message than just "AJAX error".
That could easily be combined with your suggestion:

I could also make it so that the action after an AJAX Error message is directing the user to the documentation page for Joomla Update.

We could redirect users with the 403 error to a dedicated sub-page of the docs, specifically treating that case, (hopefully) making it pretty straightforward.

@richard67 I wondered about that as well but presumed @nikosdion had a special reason for suggesting that. But maybe he just didnt know about the pre-built packages for this pr https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/35388/downloads/47158/

For testing I suggest 3 scenarios:

1. Update an unpatched 4.0-dev or latest 4.0 nightly or 4.0.2 to the patched package of this PR.
2. Update a patched 4.0-dev or latest 4.0 nightly or 4.0.2 to the patched package of this PR.
3. Update a 3.0-dev or latest 3.10 nightly or 3.10.1 to the patched package of this PR.

The 3rd test is good for having a scenario with the most things to do on update, i.e. all update SQL scripts have to be run and all these files and folders have to be removed, so from this scenario I would expect the longest execution times and the highest memory consumption.

@richard67 and then test an upgrade using the new patched version

@richard67 and then test an upgrade using the new patched version

@brianteeman Sorry, I don't understand this comment. All my 3 scenarios end with that step, upgrade using the new patched version.

Update an unpatched 4.0-dev or latest 4.0 nightly or 4.0.2 to the patched package of this PR.

end with that step, upgrade using the new patched version.

where is the step testing that this new code will update a site

@nikosdion It seems there went something wrong with updating your branch because it shows also the changes from PR #35362 which has recently been merged into the 4.0-dev branch. Maybe it helps if you rebase to 4.0-dev in your local git client and then push? Let me know if I shall try to help with that.

@richard67 @brianteeman Right. I wrote the instructions before I made the PR so I documented what I did locally. This ensures that replication is deterministic. Relying on an automatically build package which may or may not be built at the time of your test (see how long it takes for the builds to run) is not deterministic. If you find a bug and I fix it I cannot guarantee that your next test will be against the correct file.

@richard67 Your tests are irrelevant since the extraction in this case is performed by Akeeba Restore. Remember that the update is extracted by the version of Joomla Update which initiated the update.

@richard67 I did rebase to 4.0-dev a few minutes ago

@richard67 Your tests are irrelevant since the extraction in this case is performed by Akeeba Restore. Remember that the update is extracted by the version of Joomla Update which initiated the update.

Yes, you are right, silly me. Tests 1 and 3 of my scenario are using the old restore scripts. Thanks for clarifying.

OK, I force pushed the branch so it stops doing silly things. I think this is better now.

@nikosdion Now after the forced push the changes look ok on GitHub, no other unrelated changes are shown anymore.

I found bug - I update latest nightly 4.0.3-dev to latest brebuilt package, process is finished succesfully.
Then I try to reinstall Joomla core files without change update URL https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/35388/downloads/47161/pr_list.xml

Update process was going OK, little bit faster, than nightly, but it can't finish - percents are going 100+. That issue repeats when I try to upload package manually.

If any additional info or update logs are required- tell me, I will write it to this PR conversation

Hosting providers - Infinityfree (php 7.4), cloudaccess. On cloudaccess php version is 8.0.9.

@nikosdion In the mock for the "move" function the opcache of both source and destination is invalidated after the rename call if that was successful here: https://github.com/joomla/joomla-cms/pull/35388/files#diff-811902e4a22537e361d29fabd412b76563d64c1e21556796dc34bd1996baba0cR141-R147

But @PhilETaylor has implemented it differently. The opcache of the source is invalidated before the rename and the one of the destination is invalidated after the rename in any case. See e.g. here https://github.com/joomla/joomla-cms/pull/32915/files#diff-87944bb26710d00cc11ee21e5a9c9242c5fd957acade50f9eed59b84f87b563bR1634-R1636 or in this function here https://github.com/joomla/joomla-cms/blob/4.0-dev/libraries/src/Filesystem/File.php#L343 .

Which way is right or better?

@richard67 My way is right. Invalidating the file before the move doesn't make sense. Another request may load that file again before moving it is complete. This would lead to the no longer present file being cached by OPcache which means that including it would work (assuming you don't check if the file exists before trying to include it). Invalidating the OPcache after the move removes the old file from the OPcache and also tells it that the new file should be recompiled next time someone tries to access it. Granted, it's an extremely unlikely event but I see unlikely events happen every day.

@Bond97 I need some more info. Does the extraction stop at some point after 100% or does it go on forever? How far did you let it run? I am trying to understand if there is a runaway condition or if the initial size is simply not calculated correctly.

@richard67 My way is right. Invalidating the file before the move doesn't make sense. Another request may load that file again before moving it is complete. This would lead to the no longer present file being cached by OPcache which means that including it would work (assuming you don't check if the file exists before trying to include it). Invalidating the OPcache after the move removes the old file from the OPcache and also tells it that the new file should be recompiled next time someone tries to access it. Granted, it's an extremely unlikely event but I see unlikely events happen every day.

@PhilETaylor Should we do it that way too in the Filesystem library?

@richard67 My way is right. Invalidating the file before the move doesn't make sense. Another request may load that file again before moving it is complete. This would lead to the no longer present file being cached by OPcache which means that including it would work (assuming you don't check if the file exists before trying to include it). Invalidating the OPcache after the move removes the old file from the OPcache and also tells it that the new file should be recompiled next time someone tries to access it. Granted, it's an extremely unlikely event but I see unlikely events happen every day.
@PhilETaylor Should we do it that way too in the Filesystem library?

Well Im going to have to disagree with @nikosdion here. First time I think Ive disagreed.

When you delete a file you have to remove its opcache BEFORE deleting the file.

This is a well known gotcha of the opcache and confirmed by reading the source code of PHP.

We already discussed, and investigated this in #32915

https://github.com/php/php-src/blob/a6dd92f8201f1de8a1b95f4054c10f7ecc7a4fd8/ext/opcache/ZendAccelerator.c#L1318
#32915 (comment)

https://www.php.net/manual/en/function.opcache-invalidate.php#116372

Instead of removing a file from opcache that you have delete, you need to call opcache_invalidate before deleting it.

@PhilETaylor OK, that's interesting. I can't read C, I can only go by the documentation on php.net. I trust you so I'll change that.

see #32915 (comment)
#32915 (comment)
https://bugs.php.net/bug.php?id=75939

This means it is not possible to invalidate a file that is removed from the filesystem without flushing the entire opcache. This can be problematic in general and for esoteric in particular.

@nikosdion I will try again to update manually and write, will extraction stop to point or not

I think, that is goes forever. In update log doesn't appear this messages

2021-08-27T08:19:17+00:00	INFO 	update	Finalising installation.
2021-08-27T08:19:17+00:00	INFO 	update	Deleting removed files and folders.
2021-08-27T08:19:20+00:00	INFO 	update	Cleaning up after installation.

Mysql 5.7.34-37, php 7.4.8.

And I see in update log, that Joomla doesn't download new prebuilt package, until I delete previous one from tmp folder. I write new custom URL before update.

Log when I first try to update "New to new":

2021-08-27T08:35:03+00:00	INFO 	update	Update started by user Dmitriy (160). Old version is ‎4.0.3-dev+pr.35388.
2021-08-27T08:35:04+00:00	INFO    update	Downloading update file from https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/35388/downloads/47161/Joomla_4.0.3-dev+pr.35388-Development-Update_Package.zip.
2021-08-27T08:35:08+00:00	INFO 	update	File Joomla_4.0.3-dev+pr.35388-Development-Update_Package.zip downloaded.
2021-08-27T08:35:09+00:00	INFO 	update	Starting installation of new version.

Log, when I change custom URL to new prebuilt package (47168).

2021-08-27T10:31:34+00:00	INFO 	update	Update started by user Dmitriy (160). Old version is ‎4.0.3-dev+pr.35388.
2021-08-27T10:31:36+00:00	INFO 	update	File Joomla_4.0.3-dev+pr.35388-Development-Update_Package.zip downloaded.
2021-08-27T10:31:36+00:00	INFO 	update	Starting installation of new version.

In other hosting provider (iPipe) all going OK, with finalising update. I think, that issue must be solved, because some people of Joomla community uses Cloudaccess as a free hosting provider.

@Bond97 I can't reproduce your finding here on my local Linux environment. I did update a normal 4.0-dev branch without this PR applied to the custom update URL for this PR, and after that I did a reinstall core files still on that custom URL. Then I've done the same but starting with the branch of this PR. In both cases it worked for me.

So maybe it needs special environment or conditions to reproduce it?

What do you see if you go to administrator/index.php?option=com_admin&view=sysinfo and check the "Folder Permissions" tab? Is all writable except of configuration.php?

And does administrator/index.php?option=com_installer&view=warnings show any warnings?

I will later test on other environments, too, e.g. local Windows, or shared hosting with opcache on.

@Bond97 Maybe the changes made with the last commit might help to see what the problem is. Just repeat your test but with the new update package or URL. Do you get some error message at the end of the reinstall?

@richard67 ok, I will wait, until prebuilt package is available.

For two of your questions - didn't find any warnings and folder permission is all green except configuration.php

I can reproduce this locally. That's good, it means I can fix it. May take a while, we're about to have lunch.

I can reproduce this locally. That's good, it means I can fix it. May take a while, we're about to have lunch.

@nikosdion That's good news. Bon appetite!

Noting the issue about .htaccess blocking access to the new file name, would it be advisable to see if that endpoint was available before continuing with an update? Maybe do a basic sanity check to see if called it gives the "access denied" json response, (Rather than a server 403 Forbidden)? This could help catch issues before people start an upgrade, which will download the zip, and then try to extract it only to then fail because the extract.php is forbidden by .htaccess rule?

Just a thought.

do we really care for bytes when we are just a regular Joomla user?

Maybe out of scope of this PR, but I really think we should consider reusing the empty state kind of look and feel as a captive page like [mockup - not real]:

One of the historic criticisms of the upgrade page with the progress bar was that when things went wrong, there was no helpful user feedback explaining how to recover from the issue. This gives us the opportunity to change that - another idea.

@PhilETaylor I think if we use "empty state" template for it, it will be nessesary to display at least "Mbytes read" and "Mbytes write" in update process.

If regular Joomla user have an issue like slow speed drive on hosting provider or so on, just loading bar is not informative to it

I disagree with the error state. Telling people to look in the wiki means that they need to go through troubleshooting to figure it out on their own. They won't. Telling them OK, here's the five things you need to do works so much better. Remember that I am doing the former in my software and I can see exactly what the people are asking in support. If they'd actually read the troubleshooting documentation they'd have solved it on their own. When we introduced the automatic log analyser which basically told them to follow one of the troubleshooting steps in our documentation we saw the number of support requests take a nosedive.

BTW, did you miss that I modified this PR to show accurate troubleshooting information depending on what is going wrong...?

Regarding the progress state, I have no objection. However, the byte range does tell me if a. the correct update ZIP file was downloaded and b. if it's extracting correctly. I'm all for making it a footnote on the page, not remove it completely.

@PhilETaylor Let's make sure that the progress is correct and doesn't loop over, THEN I can improve the layout.

It's safe to say that right now the scope of this PR is the entire update progress page, not just the extract.php file. The goal was to modernise the updater so let's give the only part of it that wasn't touched pre-4.0 the once over. Right? :)

Anything is better than the current alert dialog that once dismissed, leaves the user on a progress bar with no other help :-) We last discussed this in 2015 here : #7612

The error screenshot was 30 seconds in inspector, it was not meant to be a final design :) If you now have better error reports that can go on that page, then great :)

I like the bytes because it shows things are moving, but Im guessing the file count also does that.

Just throwing ideas out there, like you said this whole process has not changed for many years.

About alert(), confirm() and prompt() : There's a consensus amongst ALL major browsers to remove these in the near future.
Check this intent to remove: https://groups.google.com/a/chromium.org/g/blink-dev/c/hTOXiBj3D6A/m/JtkdpDd1BAAJ

@dgrammatiko I am aware and I am also aware of the pushback. This discussion is far from being over.

In any case this is a fun topic to discuss but irrelevant to this PR since I am using a Bootstrap modal, not an alert, to display errors. Unless I missed something that I can't see?

since I am using a Bootstrap modal, not an alert, to display errors

Sorry I missed this change. Ignore me but the maintainers should have a viable plan for the removal of the few instances of alert(). Sorry for the noise

No worries! I'm spending this week on improving Joomla. Things move pretty darn fast when I have time on my hands ?

@Bond97 @richard67 You can retry now. I fixed the problem with the infinite loop during extraction.

@nikosdion There is still one place where it needs to move the File::invalidateFileCache up so it comes before the @unlink: https://github.com/joomla/joomla-cms/pull/35388/files#diff-617530abeea53181b5e5f641ba3906dcbf758238861e8a5281422722533d8532R595-R596

@nikosdion issue is fixed, I install latest nightly and update "Old to New", then "New to New". All is going OK.

I have a few questions for new empty state template

1. Maybe it will be better to write "bytes" in state "read" and write"? Just digits are not informative for regular Joomla user.
2. I think, that state "write" needs own icon.
   

"This might take a while": although on some webhosts this is correct and informative, it is still negative language, and a more positive tone could be used.

It can also be false, because on some hosts it can be really quick.

@PhilETaylor While it is obvious to you I need to say this because not everyone reading this knows who I am. I have written and continuously maintained JoomlaPack / Akeeba Backup since October 2006. I wrote Akeeba Kickstart, part of which is Akeeba Restore which is already used in Joomla Update to extract the update ZIP file. Joomla Update was contributed by me in 2012 and it's a fork of a Joomla updater feature I had written for Admin Tools since late 2010. I know better than you or anyone else that every server takes a different amount of time. That's the whole point of having Joomla Update work the way it does instead of simply using ZipArchive to extract the update in a single page load. I don't need a reminder for something that's been part of my core business for the past 16 years.

I can, however, tell you exactly why telling the user that it may NOT be instantaneous is in order.

The timing settings in extract.php are 4 seconds of maximum execution , 3 seconds of minimum execution time and 90% runtime bias. This means that the extraction will run for anywhere between 3 and 4 seconds, typically 3.6 seconds (90% of 4 seconds) give or take a few milliseconds. The code has VERY detailed comments in the constants to explain how these timing settings work for your edification and to avoid someone messing with them with non-obvious but detrimental results.

That said, this does NOT mean that everyone will see the interface update within roughly 4 seconds. You also need to add the time it takes for the request to reach the server, get handled and the response to reach the browser. In other words you will only see something on your browser in typically 4 to 5 seconds if you have a reasonably fast server and you're on a reasonably low latency connection to a server that's relatively close to you. It will be more if you're on a high latency connection (think about satellite, rural radio links and the like), the server is overwhelmed or it's across the world — or all of the above. Some users will be sitting there for 10 seconds waiting for something to happen.

At 4 seconds the user is barely noticing that it's taking a while for something to happen. At 7 to 8 seconds they're convinced it's broken. That's based on bounce data from actual sites. I did a presentation in April about that in JoomlaDay USA.

Reducing the timing parameters to give faster feedback is actually detrimental to the cause! High latency connections will only get a couple seconds shaved off the response time which may not be enough to prevent users from being concerned. Reasonably fast servers will see the update take substantially more due to the greater number of requests required. Since we are doing more requests at a far more rapid pace on these servers we exponentially increase the chance that we will hit a rate limit on the server, causing the update to fail. The timing parameters I chose are the happy medium that works on the vast majority of servers according to my 16 years of experience on the subject, stemming from having answered more than 30,000 support tickets, 10,000 emails and having troubleshooted these issues on several thousand real world servers.

Setting the expectations of the user low enough makes it far less likely that they will thing the update is broken and try to do something stupid, like hit the back button on their browser while the first request to extract the update is still running, breaking their site.

As for the tone, yes, I am aware that it's mildly negative. It's not as bad as “take several seconds” or even the blander “take some time”. Removing that altogether is not a good idea. If you think that there's a better way to write it, please do. All I have to say for myself is that even though I am not a native English speaker I acquired my Certificate of Proficiency in English at the age of 16, I am nearly 41 and I've been married for 5 years to an unapologetically pedantic American with a penchant for correcting my grammar. She did see the message and she had no objections. Considering that she's also a UX designer and led the Joomla UX team in 2015 that's good enough for me.

But, please, if you do have a suggestion then by all means please do tell me what this message should read.

It's fine by me. Far better to tell me it that it could take a while and be pleasantly surprised that it was faster than that, than wondering what went wrong because it is taking more than a few seconds and cancelling the update leaving a broken state.

And after that tirade, I'm out. Totally uncalled for rant yet again.

@PhilETaylor I am sorry that my reply came across as overly aggressive. It was honestly not my intention. The way you phrased your comment came across as insensitive and insulting. In response I was trying to clarify what was on my mind when writing that message. I was tired; more emotion than would be appropriate spilled out to that response which came across as an aggressive rant against you. I apologise.

We are both professionals with a mutual respect for each other on both a professional and a personal level. Can we please acknowledge that we both overreacted to what is in retrospect a simple misunderstanding, shake hands (or bump elbows, this being amidst a pandemic) and move on?

... I am going to convert them to KiB / MiB automatically.

@nikosdion Let us know when your PR is ready for testing. I'll wait for that in order not to waste time with testing before it's ready. Thanks in advance.

@nikosdion here's a little snippet:

const formatBytes = (bytes, decimals = 2) => {
  if (bytes === 0) return `0 ${Joomla.Text._('JLIB_SIZE_BYTES')}`;

  const k = 1024;
  const dm = decimals < 0 ? 0 : decimals;
  const sizes = [
    Joomla.Text._('JLIB_SIZE_BYTES'),
    Joomla.Text._('JLIB_SIZE_KB'),
    Joomla.Text._('JLIB_SIZE_MB'),
    Joomla.Text._('JLIB_SIZE_GB'),
    Joomla.Text._('JLIB_SIZE_TB'),
    Joomla.Text._('JLIB_SIZE_PB'),
    Joomla.Text._('JLIB_SIZE_EB'),
    Joomla.Text._('JLIB_SIZE_ZB'),
    Joomla.Text._('JLIB_SIZE_YB')
  ];

  const i = Math.floor(Math.log(bytes) / Math.log(k));

  return `${parseFloat((bytes / Math.pow(k, i)).toFixed(dm))} ${sizes[i]}`;
}

// Call it like
formatBytes(someValue, 2);

With some strings (I have no clue if those should be even translated):

JLIB_SIZE_BYTES="Bytes"
JLIB_SIZE_KB="KB"
JLIB_SIZE_MB="MB"
JLIB_SIZE_GB="GB"
JLIB_SIZE_TB="TB"
JLIB_SIZE_PB="PB"
JLIB_SIZE_EB="EB"
JLIB_SIZE_ZB="ZB"
JLIB_SIZE_YB="YB"

Code from: https://stackoverflow.com/questions/15900485/correct-way-to-convert-size-in-bytes-to-kb-mb-gb-in-javascript

@dgrammatiko That's the same snippet I am using in Akeeba Backup and the one I was planning on porting :) Good idea about the language string. Thanks!

@richard67 @Bond97 Now it's ready to test. Thank you in advance!

@nikosdion now it's almost OK.

Now need fix two things:

1. Progress bar is broken, it displayed 100%. Normally it must display 1..2..3% and so on.
   Note: I didn't do fresh install prebuilt package, maybe it's fixed already.
2. Maybe it will be better to add some wrapper to phrase "Updating your Joomla files"? For example, frame or some other thing. Or in this time it's all OK?

   Note: I didn't do fresh install prebuilt package, maybe it's fixed already.

@Bond97 How did you test then? Just applied the patch somehow? That might not be enough because it needs the js to be built. Please use the latest prebuild package.

@brianteeman Thanks. I applied your suggestions.

@Bond97 You are right about the message without a container in that it should no longer be there. I removed it now.

Regarding the progress bar, a longer explanation is in order. Even though I already explained this in my reply to Phil yesterday and in the code comments I have no problem reiterating and explicitly addressing why this happens, why it is intentional and why it should not change — if nothing else, should someone break it I can point them back here and my code comments as a “told you so”.

You only get feedback once every 3 to 4 seconds. THIS IS INTENTIONAL. It's so that the update doesn't break on the majority of servers.

If you make the feedback faster e.g. every second or so you are making far more requests per minute to the server. This will trigger the rate limiting on many commercial servers which will deny further requests from the user's IP. Not only you end up with a half-updated and broken site, you also end up impossible to access the server hosting your site to address the situation.

Even on a server without such a protection, should you increase the reporting frequency the update will take much longer because the roundtrip to the server and the time it takes to handle each request ends up being significantly more than the time spent doing productive work (extracting files).

If you make the feedback less frequent you run a greater risk of timeouts or triggering a CPU usage limit, causing the update to fail in mid-process with catastrophic results for the site.

This is a delicate balance. The values I chose are those I know from 15 years of experience and having worked on thousands of real world sites to be most likely to work on the overwhelming majority of sites. You will see that they are very close to those of Akeeba Kickstart (min/max/bias here are 3/4/90, in Kickstart they are 2/5/75, in both cases you get feedback roughly once every 3-4 seconds). The reason I used 3/4/90 instead of 2/5/75 is that I am MORE cautious in Joomla than I am in Kickstart since, unlike Kickstart, I won't be available to provide support to the very few users who have an extraction problem.

The only way around the interrelation of feedback frequency and prevention of extraction crashes is using websocket connections BUT this requires configuring your server for this i.e. it can only happen on dedicated servers. But if you are on such a server the update only takes a second and you can conceivably update it through the command line (since you'd already know enough to work on the command line, otherwise how are you managing a dedicated server?!) making the whole point moot.

Speaking of which, let's assume that there was a MAGIC way to send feedback to the interface at any frequency we want without causing any problems, just for the sake of argument. If you sent realtime updates for each of the 4500+ files being updated you would end up taking > 90% of your processing time sending updates, making the extraction dead slow. This is a problem I encountered when writing Akeeba eXtract, a desktop archive extraction application which by its very nature had no theoretical limitations in the rate of updates. Realtime updates for every file file increased the extraction time 10x to 20x and showed a blur in the interface, while hogging the CPU for the useless task of updating the interface. The best results were had at an interface update rate around once every 1-2 seconds. This brings us back to your experience. The update finished extracting in under 1.5 seconds on your servers. So, even if we had that magical capability you'd still see the progress bar jump from 0 to 100.

Note that when using Akeeba Restore you'd probably see the progress bar on these servers jump from 0 to 85 and then very quickly to 100. What can I say? 5 years after the last version I wrote much faster code ?

@Bond97 I think now the PR is ready for testing again. Could you redo your tests with clean starting conditions, i.e. not the result from previous tests, and using the new pre-built package? And clear browser cache before testing to make sure you got rid of previous versions of the js. Thanks in advance.

@brianteeman Thanks. I applied your suggestions.

Thanks

tested updating an old beta8 site to the release with this package. Everything processed smoothly.

tested updating an old beta8 site to 4.0.2 and then the release with this package. Everything processed smoothly.

When I now try to "Reinstall Core Files" everything appears to work except that there is no display of bytes or files

I test latest prebuilt package with fresh install, all works OK.

@nikosdion I agree with your thoughts, but we must think about some people, that use free hosting providers with slow servers (hdd drives or manually speed limit). It can be students or peoples with own "pet" projects.

I know, that many peoples, which have real world sites have good hosting provider accounts with fast servers or VPS/dedicates server. Update process will take 4-5 sec or less time.
But Joomla have project - free hosting with company cloudaccess. It servers some slow and update process can take 1-1,5 minutes.

My suggestion- delete percents display, only green progress bar. That maybe will be better, I think. On cloudaccess user see for 1-1,5 minutes 100 percents - it will be some not informative.

Your thoughts or suggestions? Or we leave it as it is now?

@Bond97 Let me make something clear. The progress bar tells you the update extraction progress. The final step, after you see the green 100% bar, is deleting old files. This is part of a core Joomla file (script.php in com_admin's directory) which IS NOT handled by extract.php and CAN NOT be made to follow the same process I am following in extract.php. Doing that would require a major rewrite of how Joomla removes old files after the update with massive backwards compatibility implications.

As for the progress bar itself, I have actually tested it works perfectly by changing the timing settings in extract.php to maximum execution time 1, minimum execution time 4 and runtime bias 50. This means that it only does half a second of work, sits on its butt for another 3.5 seconds and then reports the progress. The update that took 1 second to complete on my server was now taking over 12 seconds which gave me enough time to see the progress bar filling up.

Please remember that this Pull Request IS ONLY ABOUT REPLACING THE UPDATE ZIP FILE EXTRACTION, it's not to fix every issue in Joomla Update.

If I were to fix every issue I would start by removing the completely unnecessary and utterly misleading pre-installation check which I have real world reports from real world users of real world sites corroborating confirming that every single thing I was saying would go wrong two months ago does go wrong and that everything I said people would be confused about is what they are actually confused about. But some people's pride gets in the way of fixing a grave mistake so I can't touch this.

If I were to also fix everything that's wrong with Joomla Update I would also change the post-update script. This is something I couldn't do 9 years ago because it would have broken the other way of installing updates i.e. installing Joomla as an extension package of the type “file”.

If the Joomla project is interested in having someone who works with real world users on a daily basis to fix everything that is wrong with Joomla Update they know where to find me.

Until then, can we actually focus on what this PR does instead of trying to derail it, make its scope totally blurry and ultimately kill it, meaning that I wasted yet another week of my life doing solid work for Joomla that was rejected for bullshit reasons? Pretty please?

@nikosdion ok, I agree with your thoughts, thank you in advance for this clarification :)

When I now try to "Reinstall Core Files" everything appears to work except that there is no display of bytes or files

I don't get the bytes and file counters shown even when updating from the 4.0-dev branch with this PR applied to the patched package of this PR ... maybe because it goes too fast? But then I would at last expect zero values to be shown at the beginning.

The issue with the counters not updating for me and for @brianteeman in case of the reinstall core file should be fixed with my PR for @nikosdion here: nikosdion#1

@richard67 Thank you, I merged your PR to my PR (woah, this starts sounding a bit like Inception).

@Quy It's definitely "writable", see https://www.merriam-webster.com/dictionary/writable I am taking no action on that.

@nikosdion When testing with my local Linux server all went fine. But when testing with a Windows server with XAMPP I got this:

When closing that modal there is no way out except of using the back button of the browser.

The zip file has been downloaded and is present in the "tmp" folder:

Not sure yet if it's related to your PR, but it could be.

P.S.: There was nothing visible in PHP error log, webserver error log or the browser console related to that.

@brianteeman Have you been testing with a Windows server? If so, can you reproduce my issue? I have fetched the branch of this PR, made a composer install and an npm ci and then made a new installation with that. So with this installation which had the patch of this PR already applied, I've updated to the custom URL created by drone for this PR. This went well for me with Linux server but failed with Windows server.

my previous tests were on *nix
I can test again on windows and *nix tomorrow

On Sun, 29 Aug 2021 at 20:45, Richard Fath @.***> wrote:

@brianteeman https://github.com/brianteeman Have you been testing with
a Windows server? If so, can you reproduce my issue? I have fetched the
branch of this PR, made a composer install and an npm ci and then made a
new installation with that. So with this installation which had the patch
of this PR already applied, I've updated to the custom URL created by drone
for this PR. This went well for me with Linux server but failed with
Windows server.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#35388 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJ4P4M6P2IRZODYVQAFPGLT7KE37ANCNFSM5C4D6C7A
.

--
Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
https://brian.teeman.net/ http://brian.teeman.net/

Test on shared hosting (Linux) with opcache on worked for me, too. Only with Windows I have that problem.

When I apply my above suggestion https://github.com/joomla/joomla-cms/pull/35388/files#r698066015 , I come one step forward on Windows. The unzip seems to do something, and progress bar and bytes and files counter get updated, but the at the end I get:

I am in the process of setting up a Windows dev environment for Joomla 4 (as opposed to developing my extensions on Joomla 4). So far I have reproduced the issue you reported and will let you know when I have resolved it.

@nikosdion What is strange: On Linux where things seem to work, I see for a short moment text "100%" in the progress bar, and then at the end it changes to 2.5%, and the bytes and file counters change, too. On windows it ends with 100% and the failure I've mentioned above. I'm investigating by adding some debug output, but maybe you are faster.

Ah, I can tell you why you had a problem at 100%. When installing the update, the package you are installing overwrote extract.php with the version that didn't have your fix. As a result it could not find the file on Windows (due to the bug you already fixed), causing the update to fail.

I was able to prevent that from happening by making a temporary change in extract.php to skip over itself when extracting the update.

I still need to check a few things but at least we know that your previous fix does work on Windows.

I still need to check a few things but at least we know that your previous fix does work on Windows.

@nikosdion Not really. Was updating from your branch, i.e. with the patch applied, to the update URL created by drone for whit PR, which gets then the package which has also the patch applied. So there is still something wrong.

All I can tell you is that when I added 'administrator/components/com_joomlaupdate/extract.php' to setSkipFiles the problem stopped happening. The only thing this did was prevent extract.php from being extracted. The only change between the two files (and I confirmed it) was that the change for Windows hosts was not present.

If I install a manually created update from my current branch the problem doesn't exist.

So, clearly, something is wrong with the automatically built files BUT I have nothing to do with this process so I can't fix it. I can only control the code in this PR, not what some CI tool does with it.

One thing that is, however, wrong on Windows is the percentage reported. I am looking into that.

I tested this on my local computer (which is using Windows) and did not see the mentioned errors anymore. Update went well for me.

All I can tell you is that when I added 'administrator/components/com_joomlaupdate/extract.php' to setSkipFiles the problem stopped happening.

@nikosdion Here on GitHub I can't see that change. setSkipFiles is only used for restoration.php and update.php. Maybe due to a mix of local changes and review suggestions applied here on GitHub you haven't pushed all?

@joomdonation How did you test? You should have the changes of this PR applied before the update and update to a package which also contains the update.

@richard67 I updated to latest package from this PR. Then I updated again.

@richard67 I updated to latest package from this PR. Then I updated again.

@joomdonation What means updated again? Reinstall core files? Because that is all you can do when having updated to the package for the PR and wanna update again.

1. First, I downloaded the package at https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/35388/downloads/47236/Joomla_4.0.x_to_4.0.3-dev+pr.35388-Development-Patch_Package.zip . Uploaded and Updated
2. Then I set custom URL to https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/35388/downloads/47236/pr_list.xml and run the update again

I believe the second update uses the new extract.php code from this PR. Or Am I wrong?

@joomdonation Your first step links to the update package, not the new install package. What did you do with that package?

The patched package you get with the custom update URL contains the changedcode from this PR, that's right.

I test as follows:

1. I fetch the changes from this PR into a new branch:

git checkout .
git fetch upstream pull/35388/head:test-pr-35388
git checkout test-pr-35388

2. I run composer install and nmp ci.

3. I make a new installation and then in backend change to the custom uprate URL for the latest build of this PR.

4. I update.

I did that on Linux and Windows.

On Linux I get what's described here: #35388 (comment)

What is strange: On Linux where things seem to work, I see for a short moment text "100%" in the progress bar, and then at the end it changes to 2.5%, and the bytes and file counters change, too.

On Windows I get this here: #35388 (comment)

I have to check again if something went wrong so the old package from before the last corrections was used, but I think it was all right, I had removed that from the tmp folder.

@richard67 Tried again using upload and update. I monitored the console and the ajax request uses extract.php already, so I think it is working correctly. I don't see the error as in your test.

Possibly you are right and the update ended with the old package from before the last fix. Am just checking.

Possibly you are right and the update ended with the old package from before the last fix. Am just checking

Yes, that could be the reason.

@nikosdion Good news: @joomdonation and you were right. My last test on Windows was using the old package before the last correction.

Now it is same as it was on Linux: I see 100% text and 100% green in the progress bar while still some changes happen on the counters. Then comes the finalize step, and there the text in the progress bar is "2.5%" and the bar is still 100% green.

Do you want to make some fix for that % text value in the progress bar? If so, I'll wait for that before doing the next, hopefully final set of tests.

@richard67 I have identified the problem with the progress bar. It's a merge error on my part. I meant to show the % when the extraction is in progress and a fixed 100% when the extraction is done. Somewhere between that change, the bytes written and merging your PR is inverted that logic and had 100% show when the extraction is in progress and a (wrong) percentage after it's over. Fixing that logic in extract.php works great, as it should.

I am also adding an optional debug mode via a commented out by default define. If someone gets an error and you want to debug it tell them to edit administrator/components/com_joomlaupdate/extract.php and change line 11 from

//define('_JOOMLA_UPDATE_DEBUG', 1);

to

define('_JOOMLA_UPDATE_DEBUG', 1);

This will do two things:

1. Create a debug log in the site's temporary directory (by default: the tmp folder in the site's root) called joomla_update.txt
2. Prevent extract.php from being replaced during the update extraction

The debug log gives enough breadcrumbs for any developer to follow the extraction process and find out what the problem is (reading the log file backwards!). It's the same concept we're using in Akeeba Kickstart but with more verbose debugging as I'm not the one who will be reading the log file.

I'll commit and push promptly.

@nikosdion As mentioned in the above review suggestion things work for me now when updating from a 4.0 with our without the patch applied to the package of this PR. But when updating from 3.10-dev or 3.10.1 to the package of this PR, I have an issue, and @joomdonation can reproduce it, too.

I get the following when updating from 3.10.x:

When I close the error alert, nothing happens anymore.

When I then use reload page button of the browser, I get (looks strange)

and the update continues and then finishes with success, and backend looks like it should be after an update to J4.

In PHP error log I get:

PHP Fatal error:  Uncaught Error: Class 'Joomla\\CMS\\Filesystem\\File' not found in /home/richard/lamp/public_html/joomla-cms-3.10-dev/administrator/components/com_admin/script.php:7365
Stack trace:
#0 /home/richard/lamp/public_html/joomla-cms-3.10-dev/administrator/components/com_joomlaupdate/restore_finalisation.php(177): JoomlaInstallerScript->deleteUnexistingFiles()
#1 /home/richard/lamp/public_html/joomla-cms-3.10-dev/administrator/components/com_joomlaupdate/restore.php(8341): finalizeRestore()
#2 {main}
  thrown in /home/richard/lamp/public_html/joomla-cms-3.10-dev/administrator/components/com_admin/script.php on line 7365

I have tried that in several environments, and in all of them I could reproduce it.

On my local Linux I got another, different error when updating to 4.0.2 without the patch of this PR here, but on 4 other environments that update went ok.

So I'm not sure if this problem is related to this PR here. To me it looks more related to #35309 , but then it should happen with a normal 4.0.2, too, in any case.

P.S.: The above things happened with and without opcache.

That sounds like a bug in the com_admin/script.php. I have some things I need to do around the house and I'll look into it later tonight or, more realistically, tomorrow morning.

@nikosdion Thanks so far and in advance. My suggestion above you can apply, I think. It's not related to the issue but a clear thing.

@nikosdion I know where my issue comes from and I have a fix, but I don't know if it's right or not. In 3.9.x and 3.10.x, "JFile", "JFolder" and "JText" and so on are just global aliases for namespaced functions from the filesystem library. But J3 "restore_finalisation.php" checks "class-exists" for the global aliases only and created proxies only for the global aliases but not for the namespaced "File, "Folder" and "Text" when the update is started, but the script.php running later for deleting the files is the J4 one and that requires "File, "Folder" and "Text".

I have a fix for it for 3.10-dev, but I don't know if that is good, and I don't know if it works for updating from older Joomla 3.x version where the filesystem and language stuff was not namespaced yet. But it solves my problem with updating from 3.10-dev to this PR here.

Could you have a look on it? I think it might at least help you to understand what happens and to provide a better fix if mine is bad.

Here the comparison: 3.10-dev...richard67:3.10-dev-fix-class-proxies-in-restoration-for-update-to-j4

And here with ignoring whitespace differences, that might look better for review: https://github.com/joomla/joomla-cms/compare/3.10-dev...richard67:3.10-dev-fix-class-proxies-in-restoration-for-update-to-j4?w=1

So I run two tests:

1. Upgrade from 3.10.1 to the upgrade package generated by this PR, getting fatal error like @richard67 reported before
2. Upgrade from 3.10.1 to Joomla 4 nightly build, upgrade works well.

So something changes in this PR causes the upgrade from 3.10 to Joomla 4 broken. I could not be sure 100% but I think it happens because with the upgrade from 3.10.1 to the upgrade package generated by this PR, the file restore_finalisation.php is not being replaced with the file which we currently have in Joomla 4, so class \Joomla\CMS\Filesystem\File does not exist and causes fatal error (we do not have change from the PR #35309 )

But if that is the case, I wonder why we do not the the fatal error with the upgrade before #35309 added. So maybe something else. Just trying to find out.

Update: Upgrade from 3.10.1 to 4.0.0 works, too (without the namespace change from this PR #35309). So I guess what I found is wrong.

One more test: Use Joomla 3.10.1, then replace the file restore_finalisation.php with the change proposed by @richard67 3.10-dev...richard67:3.10-dev-fix-class-proxies-in-restoration-for-update-to-j4

After that, update to the upgrade package from this PR and it works. So that change might needed, but still don't understand why upgrade from 3.10.1 to 4.0 (earlier than PR #35309 ) works without fatal error.

The problem updating from Joomla 3 is what I explained in the introduction text of the PR under “Good news: you do NOT need to issue an update to Joomla Update”.

We do not include a restore_finalisation.php file since we renamed it to finalisation.php. What happens when you update from a previous version of Joomla which used Akeeba Restore is that the update takes place using the old Joomla Update which used Akeeba Restore (restore.php) to run the update and subsequently tried to load the restore_finalisation.php file to finalise the extraction.

Since we do not provide a restore_finalisation.php file the one from the previous Joomla version is used. This is what causes the problem.

The only way around it is to provide a restore_finalisation.php which is a stupid proxy to finalisation.php. HOWEVER this means that I also need to remove the code which would remove this file from the update model and rely on script.php in com_admin to do so.

The secondary problem from that is that the files installed on the server DO NOT match the list of files in the Joomla ZIP file since we delete a file we just installed. This will make security scanners flag a problem with the site (core file is missing).

So, the only way to properly handle it is to create a restore_finalisation.php file which is a proxy to finalisation.php and never remove it until such time as we decide to no longer offer an upgrade path from an old version of Joomla. In theory, this would be Joomla 6 (since you would still want to be able to update from Joomla 4.0.0-4.0.2 to Joomla 5.x). In practice, restore.php will stop working around PHP 9 since it's an ancient version last updated in 2017 and any newer versions of that file I have made are incompatible with how Joomla Update works (I have stopped using encryption as authentication since late 2017!).

I am going to commit and push a change to implement what I described. I will also comment that file so people know why it's there and why it can't be removed for another, dunno, ten years?

@brianteeman I would rather not add an extra word because plural is hard across languages and the JavaScript version of Joomla.Text does not support per-language plural strings.

Besides, this information is mostly so those of us likely to do support can make sense of what is going on. For example, if I see that the bytes in are about the size of the update ZIP file and the number of files written matches what I see in the update ZIP but the interface is stuck then the problem is something on the server throwing a hissy fit. If I see the bytes progressing and no files being extracted the problem is that someone messed with extract.php and now it's skipping over files instead of extracting them.

do you think we should add the word files here.

@nikosdion Maybe @brianteeman 's question got lost in my flood of posts (sorry for that). As far as I understand he just wanted to know your opinion.

Ah just as I posted the answer came. Sorry for me being impatient.

the JavaScript version of Joomla.Text does not support per-language plural strings.

You can use this hack:

$document->addScriptOptions(
  'someId',
  [
    'singleFile' => Text::_('JLIB_FILE'),
    'pluralFiles' => Text::_('JLIB_FILES')
  ]
);

and collect them in the browser with

const texts = Joomla.getOptions('someId');

// use them like 
// `${fileCount} ${fileCount === 0 ? texts.singleFile : pluralFiles}`

Naming is bad but you get the idea. Also I think Brian asked for this change for A11y reasons

Edit
Basically you can pass the strings to the browser as usual Text::script('JLIB_FILE') and have the ternary to pick the right one ${fileCount} ${fileCount === 0 ? Joomla.Text._('JLIB_FILE') : Joomla.Text._('JLIB_FILES')}

@nikosdion Unfortunately the Ajax error described in my comment above is still there when updating from current 3.10-dev to the package built by drone for this PR. Only the log about class not found in the PHP error log has disappeared.

The problem might be that the files and folders deletion in script.php is called 2 times, one time by the finalisation.php and one time in the model. Maybe we should not delete the "restore_finalisation.php" with the files deletion in script.php? I will check that.

No, doesn't help. And I've deleted browser cache and no opcache enabled.

Basically you can pass the strings to the browser as usual Text::script('JLIB_FILE') and have the ternary to pick the right one ${fileCount} ${fileCount === 0 ? Joomla.Text._('JLIB_FILE') : Joomla.Text._('JLIB_FILES')}

@dgrammatiko Really === 0? I.e. "0 file, 1 files, 2 files"? Maybe better === 1? But in other languages it might be different since they have endings for none, one, two and many.

@nikosdion I've even tried with a modified package which does not remove any of the old PHP and js stuff at all, neither in script.php nor in the model, but that also did not help. I still get the Ajax error. So either it really needs to make a PR for 3.10-dev, too, with my prepared branch (I have tested it and it worked), or maybe it needs some forcing the autoloader or jloader or whatever to refresh its info here?

I really don't want to delay this PR but unfortunately I have that issue.

Really === 0?

Yeah, why not? It's javascript anything goes... (obviously I meant 1)

But in other languages it might be different since they have endings for none, one, two and many.

Well there's Intl: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/PluralRules

Now, during the upgrade, there is Invalid Ajax data: (see the attached screenshot)

Now, during the upgrade, there is Invalid Ajax data:

@joomdonation You are right, and I should have watched more carefully. Before the last fix I got "AJAX loading error: Internal server error", and now with the last fix I get the same as you, "Invalid AJAX data".

@nikosdion Does that help you somehow?

@dgrammatiko The problem is that we'd need to load all the N* variants. In Greek and English we only need something like JLIB_FILES_N and JLIB_FILES_N_1. In Russian they would need JLIB_FILES_N_5. Other languages have weird rules about hundreds or thousands. The server-side Text works because it tries to find the underscore number suffix (e.g. _5 when the count is 5) in the loaded language keys when displaying the translation. Since the JS equivalent requires us explicitly pushing transactions we cannot do that unless we make changes in both the Language and the Text package, as well as the JavaScript Joomla.Text package. This is getting way out of scope for this PR and I decided not to touch it. In other words, I know exactly how to do it in both server– and client–side but I'm not touching it because I value my sanity (not too much, I am still contributing to Joomla, but I haven't gone completely crazy either).

If you feel brave, go for it, mate! When pushing a translation with Text::script you need to check for keys which start with the same subscription followed by an underscore and a numeric suffix, then push all of them. On the client side you need to modify Joomla.Text to add a plural(key, count) method which works similarly to the server-side. At-mention me if you do this and want someone to test this. I'll happily do so and I'd happily use that on my extensions!

@richard67 Sorry, my bad. I forgot to make changes in the script as well. I'll get back to that a little bit later. I'm currently doing some work on my business side of things. I need to be able to pay the bills before I can enjoy contributing to FOSS ☹️

If you feel brave, go for it, mate!

Not worth the effort.

I had forgotten to declare a constant. Doh! I made a direct edit with the fancy new VS Studio on GitHub thingie that pops up when you press dot on the keyboard since I'm on an iPad right now. Any takers for the actual test once the package builds? @joomdonation and @richard67 maybe? ?

(One of these days I need to set up a Joomla 4 dev site on a server I have SSH access to so I can test things even on an iPad and not have to bust your balls. Sorry. This is on my never ending to-do list.)

You can only reliably use plural forms in Joomla if you calculate the string on the PHP side and pass it back in the AJAX respone.
You can do that if you know the count on the server side. If you only know it on JS side, you would have to pass it as part of the AJAX request or so.

@Bakual This. won't fly in the context of extracting the Joomla Update ZIP archive since the whole premise of needing a separate extraction script is that we cannot run Joomla since it is still being extracted and might not even load at this point. So... no “Files” message is the best way to go :)

Thanks. I did not follow other discussions but in term of update, it is working well. Tested:

• Update from 3.10.1 to the update package of this PR
• Update from 4.0.0 to update package of this PR
• Tested update again after update to the package of this PR (to make sure the new update works well)

With all the 3 above tests, update was success, no errors.

@joomdonation Thank you! That was the successful test I was looking for :)

I have tested this item ✅ successfully on b77d1fe

@nikosdion So the only remaining question is how we communicate and document that the advanced .htaccess file descibed e.g. here needs a change in the " Advanced server protection rules exceptions" section: https://docs.joomla.org/Htaccess_examples_(security) . Could you create a section "Documentation Changes Required" at the end of this PR's description and mention that there? Thanks in advance.

@richard67 So... when people copied over my Master .htaccess to the Joomla docs they forgot to add any changes for Joomla Update. LOL! In fact, I see that the version of the file used is from the Joomla 1.5 times. This needs to be replaced with https://github.com/nikosdion/kyrion-htaccess/blob/kyrion/.htaccess I'll create a documentation section at the end of the PR description with both possible courses of action so you can pick what to do.

Thanks

@nikosdion @richard67 if I read correctly the above link for the htaccess, it doesn't use the pre gzipped files for Joomla 4 so it's not quite right

Well, there are many problems with the proposed .htaccess. I have also not updated my Kyrion .htaccess repository, I have only updated Admin Tools .htaccess Maker feature with the Joomla 4 changes. I will address that in the troubleshooting documentation I am writing along with all other possible problems I have seen on a multitude of live sites and based on my experience with extension updates in general. Hold tight, this will take a while...

@nikosdion @richard67 if I read correctly the above link for the htaccess, it doesn't use the pre gzipped files for Joomla 4 so it's not quite right

Yes, it’s a bit outdated and so not adapted to J4.

@richard67 I update the PR description, including a SUPER DUPER ULTRA EXTRA EXHAUSTIVE troubleshooter guide based on my experience doing that for a living for well over a decade. It may warrant its own page instead of taking over the entire Joomla Update documentation page. You are responsible adults, I trust you know how to best handle it :)

i didn't test it on a webserver on the wild
i tested it on windows with docker without WSL on a laptop

in these scenario

• Update from 3.10.1 to the update package of this PR
• Update from 4.0.0 to update package of this PR
• Tested update again after update to the package of this PR
  with
• php 8 / 7.4
• postgresql

it works fine for me

i'll not mark my test as "I have tested this item successfully "

just only for that:
"PLEASE TRY THIS ON A TEST SITE ON A COMMERCIAL HOST, IDEALLY ON A SITE THAT IS A CLONE OF A REAL WORLD SITE. DO NOT ONLY TEST ON A BLANK JOOMLA 4.0.2 SITE ON LOCALHOST. "

@alikon This does count as a test. Docker via WSL on Windows is slower than the live hosts everyone else has tested, including me. The speed of the host is a deciding factor in whether this code works so I would say that your test is very much valid and can be marked as such.

I have tested this item ✅ successfully on b77d1fe

RTC

Of course I have tested on real hosting environments, too, and on different local ones, and I have also tested the new debugging possibility in extract.php.

I've just tested again and it still works after the removal of the unnecessary reference to jQuery. So I think RTC is still valid.

So I performed some tests with opcache enabled and the result also good:

• Before this PR, I got random errors with upgrade from Joomla 3.10 to Joomla 4, and from 4.0 to nightly build
• After this PR, I tried to upgrade again and again, no error

So this PR also solves the issue with update when opcache enabled. So beside some users have to change .htaccess file as mentioned in the PR, it is all good !

Regarding the possibly necessary changes in .htaccess files with advanced protection rules for the backend, could it make sense to have a postinstall-message? If so, I could prepare a PR for that for the 4.0-dev branch here.

Beside that I see no reason why this PR should not go into 4.0.3.

I think it is a huge improvement on maintainability, stability and debug-ability of the core update process.

Thank you very much for it, @nikosdion . I hope you enjoyed working with us.

@richard67 Yes, I did enjoy working on this with you :) I like talking code and I enjoy collaborating with people to make Joomla easier.

Regarding the post-install message, I am not entirely sure if something which will only apply to a very small number of sites which have taken an action outside of what Joomla itself offers warrants a message. I think that what we already do in this PR, catching the issue and showing useful information, is far more likely to be useful and less confusing. A post-installation message is more likely to be read by people who have no clue what this all means and lead to confused posts in the forum.

I hope that made sense for you too. It made more sense in my head, I swear.

Just a quick update here - as the plan has always been to release 4.0.3 this coming tuesday - I'm not merging this immediately because we'll need to setup docs and marketing for the htaccess changes (and @nikosdion I assume on your side too we'll need to co-ordinate for Akeeba Admin Tools). My plan is to merge this next week immediately after the 4.0.3 (and given a couple of days in case of any emergency releases) which gives us a full month to put out the correct docs and marketing messages and for you to get a release out with hopefully a bit less pressure!

@nikosdion The script.php has a conflict now due to the recently merged PR #35478 . If you want I can solve that for you, just let me know. Update: I've allowed myself to solve that conflict and update the branch. I hope that was ok.

@richard67 Thank you!

@wilsonge I agree. I will make the necessary changes to Admin Tools but won't make a release just yet. I truly appreciate the extra time built into this co–ordination. We're at the second week into our daughter going to pre-school. We've already had the first weekend of all of us being varying degrees of sick with a mild respiratory tract virus she brought back from school ?

@wilsonge I agree. I will make the necessary changes to Admin Tools but won't make a release just yet. I truly appreciate the extra time built into this co–ordination. We're at the second week into our daughter going to pre-school. We've already had the first weekend of all of us being varying degrees of sick with a mild respiratory tract virus she brought back from school ?

Ouch! Hope you feel better soon and she's enjoying school :)

4.0.4 is scheduled for October 26th just so we have a timescale to work towards! Hopefully will start docs + marketing efforts on this next week once 4.0.3 has bedded in a bit.

I'm recovering very well, thanks! I am working today on my side of things. I hope the documentation I provided helps. If you need clarification on anything feel free to ask. It's a shame we never thought of writing a troubleshooting guide in the past. Better late than never, right? ?

Thanks! I'll start on the docs upgrades over the next few days so it's all there for release day

@softforge we need to get marketing kicked off on this I guess from next week.

Woo-hoo!

Is it OK if I made an Admin Tools release this week, though? I need to provide fixes for other bugs we discovered after I submitted this PR. I will make a note in the release notes that this will be available starting with Joomla 4.0.4 to prevent any misunderstandings. I can even link to this issue so the few people reading the release notes have a clue ?

OK let me try and super simplify things.

1. If you do not have an .htaccess file on your site. - stop reading this does not effect you
2. If you have never edited the .htaccess file - stop reading this does not effect you.
3. If your htaccess file does not contain this line - RewriteRule ^administrator/components/com_joomlaupdate/restore\.php$ - [L]) - stop reading this does not effect you
4. If you are still reading and you are using joomla 3 continue to point 5 else jump to point 6
5. replace the line with RewriteRule ^administrator\/components\/com_joomlaupdate\/restore\.php$ - [L] - Stop reading
6. If you are still reading and you are using joomla 4 replace the line with RewriteRule ^administrator\/components\/com_joomlaupdate\/extract\.php$ - [L] - Stop reading
7. Why are you still reading I told you to stop.

@davidascherG You make a very large of assumptions, none of which are factually correct.

99.9% of people using Joomla — and definitely everybody who is not technical — need to do ABSOLUTELY NOTHING WHATSOEVER to prepare their sites for this change. Nothing. At. All.

The only people who have to do something are those who have customised their .htaccess beyond the sample htaccess.txt file shipped with Joomla and ONLY IF their customization falls into one of the following categories:

1. a custom .htaccess file based on the .htaccess Examples (security) page in the Joomla documentation. This is based on an obsolete version of my Master .htaccess from before 2013.
2. a custom .htaccess file based on the Kyrion .htaccess (formerly Master .htaccess) I have written, i.e. the latest version of the file in the previous category.
3. a custom .htaccess file created by a third party security extension such as Admin Tools Professional.

Even then, the change is ONLY required if you have applied the advanced server protection rules for the backend which prevent access to any .php file not explicitly allowed in the .htaccess. This is literally something only very advanced users and my clients will have done. Everyone else — especially people who are not developers, systems administrators or power users — are unaffected and need to do ABSOLUTELY NOTHING AT ALL.

Nothing special is required on Windows. I don't know where you got that idea? If you are talking about IIS or NginX the only people affected are those using my software, Admin Tools Professional, to create a custom web.config or NginX configuration respectively.

Nothing is made easier for any third party developer (3PD). In fact, it does NOT affect third party developers at all either positively or negatively. This only has to do with Joomla Update, i.e. how Joomla updates itself. Updates to third party extensions are not affected at all.

The only 3PD affected is me who wrote this PR and I am negatively affected. Not only did I carry the responsibility of updating Joomla Update itself but I also had to bear the responsibility of updating Admin Tools Professional to address the change I made in the Joomla core. I created more work for myself! To make it clear, if you are using Admin Tools Professional and its .htaccess Maker / Web.config Maker / NginX Conf Maker all you need to do is update to the latest version and click the button on your screen to regenerate the .htaccess / web.config / nginx.conf file.

This means that the only people practically affected by this change is about 0.1% of expert Joomla users who maintain their own security–strengthened .htaccess file. This is the target audience of the post made by Joomla: the 0.1% of expert users.

In very simple terms, if you do not understand what to do then there is a 99.9% chance that you need to do ABSOLUTELY NOTHING.

As to why this change was made:

1. I was sick and tired of people complaining about “Akeeba taking over Joomla” just because Joomla was using a piece of code (Akeeba Restore) I wrote outside of a Joomla contribution in the Joomla Update and respected the license of my code by keeping the copyright notice, just like it does for EACH AND EVERY third party piece of code it uses. No, it makes no sense. Some people are evil like that. In any case, the code in this PR is copyrighted by OpenSourceMatters Inc, Joomla's non–profit company which legally owns the copyright to all of the core code. This makes Joomla more independent.
2. The version of my code used by Joomla was already 5 years out of date and contained a very low priority security issue I addressed four years ago. In very rare cases when Joomla Update would get stuck an attacker could gain remote code execution privileges on your site after bombarding it with a few hundred thousands requests. The code in this PR does not have this security issue anymore. This makes Joomla more secure.
3. That old code had a lot of issues with regards to error reporting and overall stability. These were fixed. This makes Joomla more stable.

I spent more than 100 hours of my own time, having my business suffer for it, to help the Joomla community. Half of that time was spent making sure nothing would break on update — a big thank you to everyone who tested and reported issues. We all made sure that the migration to the new updater backend would be seamless and would cover all sorts of upgrade cases without any work required by the overwhelming majority of Joomla users and definitely by those Joomla users who are not very technical. You're welcome, I guess...?

Unfortunately, this experience makes me wary of trying to fix anything else in Joomla Update. I was planning on helping with Joomla Update showing misleading information about third party extensions when updating to a new major release or new version family. I have already identified the problem and reported it. I was asked to fix it. Seeing how people complain about fixing what was broken for years I am not going to spend my time only to be met with hostility for fixing things. It's far easier for me to complain that something is still broken after I have reported exactly what is broken and how to fix it, let someone else spend their time to fix it and be met with hostility for their trouble. This kind of community reaction is why nobody wants to contribute to Joomla.

Hi @brianteeman Can we put that in docs and add it to SM, its a great infographic

@softforge What is “SM” (in this context)?

Social Media

No need to ask

Thank you kindly.

To those who have no idea why nick responded to a post from me that you can't see - I deleted the post in question @nikosdion within a few minutes of having written it. I have nothing but the highest respect for nick's efforts in fixing these complicated issues with Joomla Update (and with his excellent Akeeba Backup and Admin Tools extensions). I thought I had put enough disclaimers in my post to make it clear that I had not fully absorbed the issues involved and was probably misunderstanding what it seemed to me would be required of Joomla Admins. To try to avoid muddying the waters, I deleted that post but apparently not fast enough for nick to not see it.

I suppose he got a copy of it sent automatically sent to him as soon as I hit the "Comment" button. I will be much more cautious about hitting that button in the future.

Correct, I get an email notification on any Pull Request I have submitted. I do not unsubscribe from notifications after they are merged in case someone finds a critical issue we all missed during testing.

Your message was not really the reason I wrote my reply. The Joomla FB group had a long thread about this which was... let's say neither based on fact nor particularly pleasant. Seeing this spill here made me reply.

Beyond what I already commented, I'd like to add two more non–obvious points and call it a day, a week, a month and a trimester.

The leadership wrote the announcement. They ran it by me to ensure technical accuracy. Here is the entirety of my feedback, sent by email, verbatim:

The only comment I have is that it should be called the Joomla Update process, not the restore process.

I think it’s also best not to mention Admin Tools by name because some people are just looking for an excuse to moan that I somehow put all that work into Joomla Update to self–advertise; as if any rational human would put 200+ hours to get a name drop in a pre–release note which will be forgotten in a month, but you know how vile some people are. Best call it “third party Joomla security software which create or modify a site’s .htaccess file”. Those who know, know. Those who don’t know, don’t need to know either.

The technical aspect is spot on.

You can draw your own conclusions ??‍♂️

The other point is the other much less reported benefit of the new Joomla Update backend: error reporting and error resolution.

For the past nine years if something didn't work during the update extraction you'd get a JavaScript alert reading “AJAX Error”. That was it. Supremely unhelpful.

With the new backend I am telling you exactly what went went wrong. Even better, I am showing you a practical list of what you need to do to fix the problem. If you know how to use FTP to upload stuff to your server you have the technical competence required for that. For anything more complicated I tell you what to ask your host to do.

I also wrote a most detailed troubleshooting documentation for Joomla Update. Hopefully someone will put it in the docs site; I don't think I have access anymore — or at least I do not seem to have saved a login for it. It's at the top of this PR right now.

I sincerely hope that my contribution does help the community — even those people who complain without having actually used the software yet (it will only be used in the next update, 4.0.4 to 4.0.5).

On a personal note, I came to Mambo in 2004 for the code, I stayed because of the community. I consider the Joomla community to be my second extended family. I care deeply about each and every one of you, regardless of what you think of me. Like all families it's a bit dysfunctional and there's some screaming at each other at times but we still help and respect each other. That's what drives me to write software for Joomla; you're all family to me and I care about you.

That said, the recent bout of shouting did take a mental toll on me, much more than I thought it would. I need to take a short break for mental health reasons. I suppose I'll recuperate and regain my resolve before the Joomla 4.1 merge window so I can fix more Joomla bugs. Just not right now. Right now I need to spend some time with my daughter and my wife.

☮️