? ? ? Failure

User tests: Successful: Unsuccessful:

avatar wilsonge
wilsonge
15 Jul 2019

Pull Request for some of Issue #16458 .

Summary of Changes

Reworks our deleted file script onto something more rudimentary however that takes into account our npm and composer deps are no longer stored in the repo.

Things left to do

  • Make the folder paths dynamic at minimum - better to use git to build them ourselves
  • Don't trash any installation removed files
  • Don't trash search folders or files
  • Reverse order the folders list so we delete the nested directories first

Testing Instructions

This script has been used to generate the existing deleted files list - which are already usable for update - but has had a few modifications since.

Basically in the build create a tmp folder grab a copy of two joomla versions - the 3.x version goes into the a joomla390 folder and the 4.x into joomla400 folder (i picked 3.9.13 and 4.0.0 alpha 12) to generate the list in this PR). Then run the script - it should match what's in the com_admin script

Documentation Changes Required

n/a

avatar wilsonge wilsonge - open - 15 Jul 2019
avatar wilsonge wilsonge - change - 15 Jul 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 15 Jul 2019
Category Repository
avatar wilsonge wilsonge - change - 15 Jul 2019
The description was changed
avatar wilsonge wilsonge - edited - 15 Jul 2019
avatar wilsonge wilsonge - change - 15 Jul 2019
The description was changed
avatar wilsonge wilsonge - edited - 15 Jul 2019
avatar wilsonge wilsonge - change - 15 Jul 2019
Labels Added: ? ?
avatar wilsonge wilsonge - change - 11 Nov 2019
Labels Added: ? ?
avatar joomla-cms-bot joomla-cms-bot - change - 11 Nov 2019
Category Repository Administration com_admin Repository
avatar wilsonge wilsonge - change - 11 Nov 2019
The description was changed
avatar wilsonge wilsonge - edited - 11 Nov 2019
avatar wilsonge wilsonge - change - 11 Nov 2019
The description was changed
avatar wilsonge wilsonge - edited - 11 Nov 2019
avatar wilsonge wilsonge - change - 11 Nov 2019
The description was changed
avatar wilsonge wilsonge - edited - 11 Nov 2019
avatar wilsonge
wilsonge - comment - 11 Nov 2019

Pushed a new tweak up - a horrible version of doing a iterator filter (there must be a better way I can deal with the excluded folders other than the prepending which requires separate filters) - but at least it works. This means com_search and installation directories are excluded and there's a simple mechanism for this in the future.

I've also reversed the order of the directories so we delete subdirectories before parent directories

avatar wilsonge wilsonge - change - 11 Nov 2019
The description was changed
avatar wilsonge wilsonge - edited - 11 Nov 2019
avatar wilsonge wilsonge - change - 11 Jan 2020
Labels Removed: ? ?
avatar wilsonge
wilsonge - comment - 25 Jan 2020

Just used this to update the latest script.php in 4.0-dev so will be able to use that as a metric of how stable this script is beyond the requirement to be able to download the two branches (for now i just downloaded and unzipped the latest nightlies of 3.10 and 4.0)

avatar SharkyKZ
SharkyKZ - comment - 25 Jan 2020

How to use this script? I tried php deleted_file_check.php --from joomla390 --to joomla400 get Missing ending directory due to extra colon here

$options = getopt('', array('from:', 'to::'));
.

avatar wilsonge
wilsonge - comment - 26 Jan 2020

php build/deleted_file_check.php --from=/Users/george/Downloads/Joomla_3.10.0-dev-Development-Full_Package --to=/Users/george/Downloads/Joomla_4.0.0-beta1-dev-Development-Full_Package

That's what I did - where the directories are the two nightlies.

I haven't changed that line from the working J3 version :/ so is this script working for you there?

avatar SharkyKZ
SharkyKZ - comment - 26 Jan 2020

That works, thanks. I haven't used this script before.

Should templates be deleted? They can contain custom overrides and assets.

avatar wilsonge
wilsonge - comment - 27 Jan 2020

Currently the intention was yes. Because we’re not shipping bootstrap 2 in j4 so they will be broken by default. However I’m willing to listen on it. If people don’t like that decision it’s easy to add it to the exclusion list

avatar mbabker
mbabker - comment - 27 Jan 2020

Currently the intention was yes. Because we’re not shipping bootstrap 2 in j4 so they will be broken by default. However I’m willing to listen on it. If people don’t like that decision it’s easy to add it to the exclusion list

Lookup the 2.5 to 3.0 scripts. IIRC the admin template was fully uninstalled but the frontend template wasn't. The same should be done for 3.x to 4.0.

avatar wilsonge
wilsonge - comment - 28 Jan 2020

The 2.5 template didn’t break I don’t think because it had no framework under it? Not 100% on that though

avatar mbabker
mbabker - comment - 28 Jan 2020

Protostar has its own CSS, it’s not using Bootstrap from the media directory. So that part isn’t busted. The JavaScript integrations probably are though, not much that can be done for that without a compatibility plugin overloading JHtmlBootstrap to restore BS2 (the same as my old BS3 demo plugin). Likewise layouts are busted because of the insistence of the default layouts being designed against the default frontend template and not being agnostic.

Still it’s probably be a bad idea to flat out wipe out someone’s template.

avatar brianteeman
brianteeman - comment - 28 Jan 2020

2.5 site and admin templates survived after the update

avatar richard67
richard67 - comment - 8 Feb 2020

@wilsonge For me this is at least a release blocker due to following reason:

We want to have updateability from beta-n to beta-(n-1).

So latest with beta-2 we need it to be included into our build procedure so the script.php is up to date for each build.

If we want to go one step beyond and support updateability between nightly builds beginning with beta-1, then it even could be a beta-blocker.

Which of these 2 labels should I set? Feedback very welcome.

avatar wilsonge
wilsonge - comment - 8 Feb 2020

I've added an explicit release blocker - but the root issue it's partly solving is already a release blocker.

I don't think it's a beta blocker as we can already use this script - it's just about making it pretty to use

avatar richard67
richard67 - comment - 8 Feb 2020

@wilsonge I think it should run automatically on every nightly build at the end.

avatar wilsonge
wilsonge - comment - 8 Feb 2020

It can not run automatically - its building array's to paste into script.php. unless we made these contents a standalone file - but that's out of scope of this PR. I'm manually putting the results into script.php right now (as was the case in 3.x too)

avatar richard67
richard67 - comment - 8 Feb 2020

@wilsonge I thought about making an RFC for using a standalone text file, or 2, 1 for files and 1 for folders, before I posted here. Yes is out of scope of this PR. But if you see a chance for such solution we find maybe time for talk on Glip what could be done how. I am not so familiar with the build process.

avatar wilsonge
wilsonge - comment - 10 Mar 2020

Happy to consider standalone files (obviously would still need a die statement at the top to stop public access but ?‍♂ )

avatar wilsonge wilsonge - change - 18 Mar 2020
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - change - 18 Mar 2020
Category Repository Administration com_admin Repository
avatar jwaisner jwaisner - change - 12 Jun 2020
Priority Medium Urgent
avatar richard67
richard67 - comment - 3 Mar 2021

Now after PR #31804 has been merged, it needs to make sure that files to be renamed are not added to the list of files to be deleted.

It might also make sense to generate a list of files to be renamed, so we have something like a renamed_files.txt.

Finally I think it always needs to run the script here in 2 steps:

  1. Compare a 4.0 Beta 1 with latest 3.10 Alpha or Beta or whatever release (or maybe even the latest 3.10 nightly).
  2. Compare the last 4.0 nightly or a just built one with the last 4.0 release for which the lists have been updated before.

Step 1 might e.g. result in new update SQL scripts for 3.10 which have to be removed when updating to 4.

It needs 2 steps because when comparing latest 4.0 with latest 3.10 only, you miss all those files having been added for 4.0 e.g. with Beta 1 and then later been removed with a later Beta.

avatar richard67
richard67 - comment - 4 Mar 2021

Hmm, it seems my above suggestion is also not the finally correct one. E.g. file /administrator/components/com_admin/postinstall/htaccesssvg.php has been added to 3.10-dev but what not present yet in 4.0 Beta 1, so we would add that file to the list of files to be deleted when comparing those 2 releases.

So it needs to think further ?

avatar dgrammatiko
dgrammatiko - comment - 4 Mar 2021

but what not present yet in 4.0 Beta 1

There's no travel machine yet AFAIK ?. What I mean is that you can't patch anything to a previous release, so the diff needs to be between 3.10 and the head of the J4 branch. So, the J4 needs a script that will clean up all the files (esp the static assets) from beta1 till the current head and also diff the latest 3.10 release. The previous betas are just broken regarding the removal of some files (esp static assets). In sort let's not complicate things just for shake of writing code. My 2c

avatar richard67
richard67 - comment - 4 Mar 2021

In sort let's not complicate things just for shake of writing code. My 2c

That's not my intention. I just want to point out things I find so they are not forgotten.

avatar dgrammatiko
dgrammatiko - comment - 4 Mar 2021

I just want to point out things I find so they are not forgotten.

Sure, I think that the only viable solution can only diff the tip of the head the 3.10 with the tip of the head of j4. What happens between beta1 till beta7 (or whatever was the latest release) should be handled as a one-off task (run it once, fix things, delete the script!).

avatar richard67
richard67 - comment - 4 Mar 2021

Especially in case of 4.0 we have to build an installation package or to run composer install with the no-dev option and npm ci, it's not enough to check just the sources.

avatar dgrammatiko
dgrammatiko - comment - 4 Mar 2021

Especially in case of 4.0 we have to build an installation package or to run composer install with the no-dev option and npm ci, it's not enough to check just the sources.

Thankfully Github actions can do that for us automatically, and merge back a file (eg /build/files-diff.json). Then the existing script.php has to only parse this file and add the entries to delete file/folder or rename. Actually it would be a good idea to have this data as a json file instead of arrays in a script file. It's easier to automate the additions, removals of entries programmatically

Eg this file: https://github.com/dgrammatiko/sloth-pkg/blob/main/scripts/init.js already has the logic to clone locally (ok in the github actions doker image) any branch. Once it's cloned run composer/npm on j4 and diff the folders. Store the array of files to a file. Have an action to merge back the diff.json. Then patch the script to pick up that data...

avatar richard67
richard67 - comment - 4 Mar 2021

Actually it would be a good idea to have this data as a json file instead of arrays in a script file. It's easier to automate the additions, removals of entries programmatically

100% agree. It's a pain to maintain the script.php file with these gazillions of lines for these 2 hardcoded monster arrays in it.

avatar richard67
richard67 - comment - 4 Mar 2021

If we want to detect files which have changed only in case (upper or lowercase mix) of their basename so that they are not added to the list of files to be deleted but to a list of files to be renamed, and if we don't want to limit successful execution of this to a particular OS, e.g. Linux, then we might have to do similar tricks like here to make it safe for any kind of OS:

https://github.com/joomla/joomla-cms/blob/4.0-dev/administrator/components/com_admin/script.php#L7176-L7218

The reason is that on Windows you can find out what is the actual name of the file, but on iOS you get it in the casing like you ask for it, except when using scandir in PHP in that case, and with a check if the 2 different names have the same inode (like on iOS) or not (i.e. really different files on *nix).

avatar dgrammatiko
dgrammatiko - comment - 4 Mar 2021

except when using scandir in PHP in that case

My solution will be Node based, so this won't be a problem. Also since the idea is to automate this with Github actions the OS will be Linux so safe for these operations.

avatar richard67
richard67 - comment - 4 Mar 2021

Sounds all very good to me. I am not good in JS, but in Git and a few other things, so if I can help with something just let me know.

In the mean time we need to continue with maintaining script.php either with the WiP here or "manually", i.e. we should not do nothing with it while waiting for the good solution. I can help with that, too.

avatar dgrammatiko
dgrammatiko - comment - 4 Mar 2021

Actually you can help by moving the array of files to a JSON file. A structure that should be ok:

{
  "forRemoval": {
    "files": {
      "3.9.10": [
        "administrator/something/file.php"
      ]
    },
    "folders": {
      "3.2.0": [
        "images/stock"
      ]
    }
  },
  "forRename": {
    "files": {
      "3.9.10": {
        "from": [
          "administrator/something/otherFile.php"
        ],
        "to": [
          "administrator/something/otherfile.php"
        ]
      }
    },
    "folders": {
      "3.2.0": {
        "from": [
          "administrator/Component"
        ],
        "to": [
          "administrator/component"
        ]
      }
    }
  }
}
avatar dgrammatiko
dgrammatiko - comment - 4 Mar 2021

Edit, maybe keep each release to it's own file, similar to what we do for the SQL files. Eg a file named 3.9.10.json with some content like:

{
  "forRemoval": {
    "files":  [
        "administrator/something/file.php"
      ],
    "folders": [
        "images/stock"
      ]
  },
  "forRename": {
    "files": {
      "from": [
        "administrator/something/otherFile.php"
      ],
      "to": [
        "administrator/something/otherfile.php"
      ]
    },
    "folders": {
      "from": [
        "administrator/Component"
      ],
      "to": [
        "administrator/component"
      ]
    }
  }
}

Then in the script.php scan the dir (let's say named redundant) read each file and json parse it. Loop the json and create arrays for flies and folders to be removed, renamed. Then map the arrays to the particular function (delete, rename). Joba done. This is also maintainable...

avatar dgrammatiko
dgrammatiko - comment - 4 Mar 2021

FWIW I just setup a rough CI https://github.com/dgrammatiko/jdiff with gihub actions and get the redundant files: https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json

Forking this all you have to do is run the action and copy-paste the redundant.json (I mean if we go with the idea of moving the data to json files)

avatar dgrammatiko
dgrammatiko - comment - 18 Mar 2021

@wilsonge @richard67 how can I verify that the collection of files https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json and folders https://github.com/dgrammatiko/jdiff/blob/main/RedundantFolders.json is what we expect to be for the J3 to J4 upgrade?

avatar richard67
richard67 - comment - 18 Mar 2021

@wilsonge @richard67 how can I verify that the collection of files https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json and folders https://github.com/dgrammatiko/jdiff/blob/main/RedundantFolders.json is what we expect to be for the J3 to J4 upgrade?

By comparing 4.0 Beta 1 with the latest nightly of 3.10-dev.

avatar dgrammatiko
dgrammatiko - comment - 18 Mar 2021

Sure, but are there files that although it shouldn't survive the upgrade we want to keep around (eg protostar, beez templates)

avatar richard67
richard67 - comment - 18 Mar 2021

Sure, but are there files that although it shouldn't survive the upgrade we want to keep around (eg protostar, beez templates)

I think the templates are deleted ... about which other files to keep I don't know. We might really have to wait for George for an answer to that.

avatar dgrammatiko
dgrammatiko - comment - 18 Mar 2021

Ok, anyways on the https://github.com/dgrammatiko/jdiff whenever you create and merge a PR will create the diffs for files and folders (based on what EXISTS on J4 and J3.10 latest branches.

avatar richard67
richard67 - comment - 18 Mar 2021

I'll have a look on it but it may take some time.

avatar wilsonge wilsonge - change - 20 Apr 2021
Labels Added: ?
Removed: ?
avatar wilsonge wilsonge - change - 27 Jun 2021
Labels Added: ?
Removed: ?
avatar wilsonge wilsonge - change - 2 Jul 2021
Labels Added: ?
Removed: ?
avatar wilsonge wilsonge - change - 13 Jul 2021
Labels Added: ?
Removed: ?
avatar wilsonge wilsonge - change - 13 Jul 2021
Status Pending Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2021-07-13 18:22:26
Closed_By wilsonge
avatar wilsonge wilsonge - close - 13 Jul 2021
avatar wilsonge wilsonge - merge - 13 Jul 2021

Add a Comment

Login with GitHub to post a comment