User tests: Successful: Unsuccessful:
Pull Request for some of Issue #16458 .
Reworks our deleted file script onto something more rudimentary however that takes into account our npm and composer deps are no longer stored in the repo.
This script has been used to generate the existing deleted files list - which are already usable for update - but has had a few modifications since.
Basically in the build create a tmp folder grab a copy of two joomla versions - the 3.x version goes into the a joomla390
folder and the 4.x into joomla400
folder (i picked 3.9.13 and 4.0.0 alpha 12) to generate the list in this PR). Then run the script - it should match what's in the com_admin script
n/a
Status | New | ⇒ | Pending |
Category | ⇒ | Repository |
Labels |
Added:
?
?
|
Labels |
Added:
?
?
|
Category | Repository | ⇒ | Administration com_admin Repository |
Labels |
Removed:
?
?
|
Just used this to update the latest script.php in 4.0-dev so will be able to use that as a metric of how stable this script is beyond the requirement to be able to download the two branches (for now i just downloaded and unzipped the latest nightlies of 3.10 and 4.0)
How to use this script? I tried php deleted_file_check.php --from joomla390 --to joomla400
get Missing ending directory
due to extra colon here
joomla-cms/build/deleted_file_check.php
Line 39 in 9d216a2
php build/deleted_file_check.php --from=/Users/george/Downloads/Joomla_3.10.0-dev-Development-Full_Package --to=/Users/george/Downloads/Joomla_4.0.0-beta1-dev-Development-Full_Package
That's what I did - where the directories are the two nightlies.
I haven't changed that line from the working J3 version :/ so is this script working for you there?
That works, thanks. I haven't used this script before.
Should templates be deleted? They can contain custom overrides and assets.
Currently the intention was yes. Because we’re not shipping bootstrap 2 in j4 so they will be broken by default. However I’m willing to listen on it. If people don’t like that decision it’s easy to add it to the exclusion list
Currently the intention was yes. Because we’re not shipping bootstrap 2 in j4 so they will be broken by default. However I’m willing to listen on it. If people don’t like that decision it’s easy to add it to the exclusion list
Lookup the 2.5 to 3.0 scripts. IIRC the admin template was fully uninstalled but the frontend template wasn't. The same should be done for 3.x to 4.0.
The 2.5 template didn’t break I don’t think because it had no framework under it? Not 100% on that though
Protostar has its own CSS, it’s not using Bootstrap from the media directory. So that part isn’t busted. The JavaScript integrations probably are though, not much that can be done for that without a compatibility plugin overloading JHtmlBootstrap to restore BS2 (the same as my old BS3 demo plugin). Likewise layouts are busted because of the insistence of the default layouts being designed against the default frontend template and not being agnostic.
Still it’s probably be a bad idea to flat out wipe out someone’s template.
2.5 site and admin templates survived after the update
@wilsonge For me this is at least a release blocker due to following reason:
We want to have updateability from beta-n to beta-(n-1).
So latest with beta-2 we need it to be included into our build procedure so the script.php is up to date for each build.
If we want to go one step beyond and support updateability between nightly builds beginning with beta-1, then it even could be a beta-blocker.
Which of these 2 labels should I set? Feedback very welcome.
I've added an explicit release blocker - but the root issue it's partly solving is already a release blocker.
I don't think it's a beta blocker as we can already use this script - it's just about making it pretty to use
It can not run automatically - its building array's to paste into script.php. unless we made these contents a standalone file - but that's out of scope of this PR. I'm manually putting the results into script.php right now (as was the case in 3.x too)
@wilsonge I thought about making an RFC for using a standalone text file, or 2, 1 for files and 1 for folders, before I posted here. Yes is out of scope of this PR. But if you see a chance for such solution we find maybe time for talk on Glip what could be done how. I am not so familiar with the build process.
Happy to consider standalone files (obviously would still need a die statement at the top to stop public access but
Labels |
Added:
?
|
Category | Repository Administration com_admin | ⇒ | Repository |
Priority | Medium | ⇒ | Urgent |
Now after PR #31804 has been merged, it needs to make sure that files to be renamed are not added to the list of files to be deleted.
It might also make sense to generate a list of files to be renamed, so we have something like a renamed_files.txt
.
Finally I think it always needs to run the script here in 2 steps:
Step 1 might e.g. result in new update SQL scripts for 3.10 which have to be removed when updating to 4.
It needs 2 steps because when comparing latest 4.0 with latest 3.10 only, you miss all those files having been added for 4.0 e.g. with Beta 1 and then later been removed with a later Beta.
Hmm, it seems my above suggestion is also not the finally correct one. E.g. file /administrator/components/com_admin/postinstall/htaccesssvg.php
has been added to 3.10-dev but what not present yet in 4.0 Beta 1, so we would add that file to the list of files to be deleted when comparing those 2 releases.
So it needs to think further
but what not present yet in 4.0 Beta 1
There's no travel machine yet AFAIK
In sort let's not complicate things just for shake of writing code. My 2c
That's not my intention. I just want to point out things I find so they are not forgotten.
I just want to point out things I find so they are not forgotten.
Sure, I think that the only viable solution can only diff the tip of the head the 3.10 with the tip of the head of j4. What happens between beta1 till beta7 (or whatever was the latest release) should be handled as a one-off task (run it once, fix things, delete the script!).
Especially in case of 4.0 we have to build an installation package or to run composer install
with the no-dev
option and npm ci
, it's not enough to check just the sources.
Especially in case of 4.0 we have to build an installation package or to run composer install with the no-dev option and npm ci, it's not enough to check just the sources.
Thankfully Github actions can do that for us automatically, and merge back a file (eg /build/files-diff.json). Then the existing script.php has to only parse this file and add the entries to delete file/folder or rename. Actually it would be a good idea to have this data as a json
file instead of arrays in a script file. It's easier to automate the additions, removals of entries programmatically
Eg this file: https://github.com/dgrammatiko/sloth-pkg/blob/main/scripts/init.js already has the logic to clone locally (ok in the github actions doker image) any branch. Once it's cloned run composer/npm on j4 and diff the folders. Store the array of files to a file. Have an action to merge back the diff.json. Then patch the script to pick up that data...
Actually it would be a good idea to have this data as a
json
file instead of arrays in a script file. It's easier to automate the additions, removals of entries programmatically
100% agree. It's a pain to maintain the script.php file with these gazillions of lines for these 2 hardcoded monster arrays in it.
If we want to detect files which have changed only in case (upper or lowercase mix) of their basename so that they are not added to the list of files to be deleted but to a list of files to be renamed, and if we don't want to limit successful execution of this to a particular OS, e.g. Linux, then we might have to do similar tricks like here to make it safe for any kind of OS:
The reason is that on Windows you can find out what is the actual name of the file, but on iOS you get it in the casing like you ask for it, except when using scandir in PHP in that case, and with a check if the 2 different names have the same inode (like on iOS) or not (i.e. really different files on *nix).
except when using scandir in PHP in that case
My solution will be Node based, so this won't be a problem. Also since the idea is to automate this with Github actions the OS will be Linux so safe for these operations.
Sounds all very good to me. I am not good in JS, but in Git and a few other things, so if I can help with something just let me know.
In the mean time we need to continue with maintaining script.php either with the WiP here or "manually", i.e. we should not do nothing with it while waiting for the good solution. I can help with that, too.
Actually you can help by moving the array of files to a JSON file. A structure that should be ok:
{
"forRemoval": {
"files": {
"3.9.10": [
"administrator/something/file.php"
]
},
"folders": {
"3.2.0": [
"images/stock"
]
}
},
"forRename": {
"files": {
"3.9.10": {
"from": [
"administrator/something/otherFile.php"
],
"to": [
"administrator/something/otherfile.php"
]
}
},
"folders": {
"3.2.0": {
"from": [
"administrator/Component"
],
"to": [
"administrator/component"
]
}
}
}
}
Edit, maybe keep each release to it's own file, similar to what we do for the SQL files. Eg a file named 3.9.10.json
with some content like:
{
"forRemoval": {
"files": [
"administrator/something/file.php"
],
"folders": [
"images/stock"
]
},
"forRename": {
"files": {
"from": [
"administrator/something/otherFile.php"
],
"to": [
"administrator/something/otherfile.php"
]
},
"folders": {
"from": [
"administrator/Component"
],
"to": [
"administrator/component"
]
}
}
}
Then in the script.php scan the dir (let's say named redundant
) read each file and json parse it. Loop the json and create arrays for flies and folders to be removed, renamed. Then map the arrays to the particular function (delete, rename). Joba done. This is also maintainable...
FWIW I just setup a rough CI https://github.com/dgrammatiko/jdiff with gihub actions and get the redundant files: https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json
Forking this all you have to do is run the action and copy-paste the redundant.json (I mean if we go with the idea of moving the data to json files)
@wilsonge @richard67 how can I verify that the collection of files https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json and folders https://github.com/dgrammatiko/jdiff/blob/main/RedundantFolders.json is what we expect to be for the J3 to J4 upgrade?
@wilsonge @richard67 how can I verify that the collection of files https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json and folders https://github.com/dgrammatiko/jdiff/blob/main/RedundantFolders.json is what we expect to be for the J3 to J4 upgrade?
By comparing 4.0 Beta 1 with the latest nightly of 3.10-dev.
Sure, but are there files that although it shouldn't survive the upgrade we want to keep around (eg protostar, beez templates)
Sure, but are there files that although it shouldn't survive the upgrade we want to keep around (eg protostar, beez templates)
I think the templates are deleted ... about which other files to keep I don't know. We might really have to wait for George for an answer to that.
Ok, anyways on the https://github.com/dgrammatiko/jdiff whenever you create and merge a PR will create the diffs for files and folders (based on what EXISTS on J4 and J3.10 latest branches.
I'll have a look on it but it may take some time.
Labels |
Added:
?
Removed: ? |
Labels |
Added:
?
Removed: ? |
Labels |
Added:
?
Removed: ? |
Labels |
Added:
?
Removed: ? |
Status | Pending | ⇒ | Fixed in Code Base |
Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2021-07-13 18:22:26 |
Closed_By | ⇒ | wilsonge |
Pushed a new tweak up - a horrible version of doing a iterator filter (there must be a better way I can deal with the excluded folders other than the prepending which requires separate filters) - but at least it works. This means com_search and installation directories are excluded and there's a simple mechanism for this in the future.
I've also reversed the order of the directories so we delete subdirectories before parent directories