I have diffs up to beta 7 (https://github.com/dgrammatiko/JfixMediaFolder/blob/main/diff_beta_6_7.json) and those are inline with your results. For the diff with the tip of the head I need to alter the script a bit (will do it tomorrow)

Can we also get a decision on the data separation? This script.php is becoming unreadable...

Can we also get a decision on the data separation? This script.php is becoming unreadable...

@dgrammatiko I'm not the guy to decide anything, that's George, I think.

I am in favour of data separation, but I'm not sure if we will have it ready before the next Beta.

So I thought I have to make a quick fix to be sure the next Beta will be ok.

@dgrammatiko I did use George's script in the following way to make this PR:

• I've used full installation packages for the comparisons.
• I've compared last nightly build of 4.0-dev with the one of 3.10-dev to get the changes from 3 to 4. I've noticed that comparing 3.10-dev with 4.0 Beta 1 like I thought once that it could be done would give false results, because new things added in 3.x and later being merged up into 4.0-dev will be added to the lists for removal when being compared with a 4.0 Beta 1.
• The result of the comparison from 3.10-dev to 4.0-dev I have added to the first section "// Joomla 4 Beta 1" in script.php.
• From this section I have removed duplicated which would be in the array because later being removed again with a comparison between 4.0 Beta x and 4.0 Beta (x+1). This is not really necessary, they only cause a bit longer run time, but they would become more and more if not removed. I have decided to remove duplicates in the 1st section "// Joomla 4 Beta 1", except of 3.x update SQL scripts, those I handle there and not in later sections.
• The later sections "// Joomla 4 Beta 1" and so on are easy. My results are as far as I could see the same as you have, but I haven't checked every detail yet.
• If not all has been merged up from 3.10-dev to 4.0-dev, it might be necessary to manually remove files or folders from the list, or to limit the comparison between last 4.0 Beta and the 4.0-dev branch to the last commit which the 3.10-dev and the 4.0-dev branches have in common, or something like that.

On a quick check, you have 4515 entries (5011 -496) for the 3x to 4.0-beta1 and I have 4833 files https://github.com/dgrammatiko/jdiff/blob/main/Redundant.json and 1167 folders (you have 1171) https://github.com/dgrammatiko/jdiff/blob/main/RedundantFolders.json (the folders are off because the script is echoing all the nested folders) so I guess we are on the same boat here. A small detail my script is diffing the tip of 3.10 against the tip of 4.0-dev which is not ideal in this case

@wilsonge can you decide on the data separation here? It would make the checks here so much easier if we have to compare xmls or jsons... #32715 is already using XML files so the only question is if we want multiple files or the build tools could append to one existing file instead

On a quick check, you have 4515 entries (5011 -496) for the 3x to 4.0-beta1 and I have ...

@dgrammatiko What I was trying to explain above is that we shall not compare 3.x to 4.0-beta1, because they are 2 branches and things which are merged up go into the last 4.0 comparison, too. It is better if we compare 3.x with the end point of the 4.0 comparisons, e.g. current 4.0-dev or last nightly. For this reason I could have changed the comments "// Joomla 4.0 Beta 1" in script.php to "// From 3.x to current 4.0" or so, but I was not sure if this might confuse people.

@dgrammatiko You should also check wilsonge#60 and wilsonge#61 . We need that in your scripts, too:

• Update exclusion list by sample data images
• Handle list of files to be renamed (which is not the same as moving. Rename is just changing the base name, and here only regarding casing. Moving we don't need to handle.)

What I was trying to explain above is that we shall not compare 3.x to 4.0-beta1

Ok, so we are doing the same diffing then. If you have the raw results can you put them on a gist, it would be easier to check against my scripts, although this shouldn't be a code validation rather a consistency check (what needs to stay or not) which tbh I don't have the complete oversight on many parts

Handle list of files to be renamed

We can add another section in the xmls, eg:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<files>
  <filename>components/com_config/src/Model/CmsModel.php</filename>
  <folder>components/com_config/src/Model</folder >
  <rename from="media/vendor/skipto/js/skipTo.js" to="media/vendor/skipto/js/skipto.js"></rename>
</files>

https://github.com/dgrammatiko/JfixMediaFolder/blob/main/diff_beta_1_2.xml

@dgrammatiko Get my raw files plus my modified PHP script and my shell script as Zip from here: http://test5.richard-fath.de/j4-update-script-deleted-files.zip . Let me know if you got it so I can remove it from there.

@dgrammatiko I have a new version of the PHP and the shell script and my raw files: http://test5.richard-fath.de/j4-update-script-deleted-files_2021-04-19.zip . My latest changes on this PR here are based on that. When comparing J4 versions, files and folders will only be added to the lists if they are not already present in the lists from the 3.10 to current 4.0 comparison. This means I could use the raw files to make up the sections in script.php without making any manual modifications (except of removing stuff from the lists which is on the way being merged up from 3.10 to 4.0 soon).

@richard67 so the idea is to keep a registry (some ref files) in the repo? If so maybe it worth keeping the data in a .json file instead of .txt. After a quick check it seems fine to me. Maybe we should finally have a decision on the data/logic separation to finalise this whole part, so @wilsonge it's up to you I guess...

@richard67 so the idea is to keep a registry (some ref files) in the repo?

Not necessarily, but it could make sense.

@dgrammatiko @wilsonge I've just updated this PR to the latest upmerges so it is up to date and ready for the next beta or RC release. You can find the raw files, my modified PHP script and the shell script which I've used in that zip: http://test5.richard-fath.de/j4-update-script-deleted-files_2021-05-03.zip . Just check the shell script and you will see how it worked. The modified PHP script contains a change for an optional new parameter to check the files from the 3.10-to-4 comparison in addition and if something is in that file already, it will not be added to any of the 4.0-x-to-4.0-y files.

P.S.: I did not have to make any modifications on the raw files. I could copy their content as is to the corresponding sections in scrip.php. This worked because after the recent upmerge, the the 4.0-dev branch is fully up to date with the 3.10-dev branch.

Thankyou!

Thank you, too.