I'm offline now for 48 hours

When testing you should ensure that the contents of the media folder are identical locally and in the download package

A review from you and feedback on whether this is what we want would be great @Fedik @dgrammatiko

LGTM

what happens if you run multiple times npm run build:css?

Good idea for testing :), I've now run it a few times in a row. Everything is working fine for me. I hope someone else will test it too, of course.

I have tested this item 🔴 unsuccessfully on 8b064e6

Thank you @LadySolveig and I tried to test it, but there are some things I don't know (⚠️) and some things are certainly wrong. I can test again when a new version is available. Please leave a comment if the test approach is not valid or needs to be expanded.

Tested with JBT recreate 5.3.4-rc1

• Using PHP 8.4.7, Composer version 2.8.1, node v22.19.0 and npm 10.9.3
• Before the PR
  • Running npm ci and safed a before-copy of the media folder
  • Seen media/vendor/bootstrap/css/bootstrap.min.css contains the version info
  • Running php build/build.php --exclude-gzip --exclude-zstd
  • Unpacked media folder from built Joomla_5.3.4-rc1-Release_Candidate-Full_Package.zip
    • Seen media/vendor/bootstrap/css/bootstrap.min.css is missing the version info
• Applying the PR with gh pr checkout 46146
  • Running npm ci and safed a after-copy of the media folder
  • Compared media folders, the after PR changes are:
    • ⚠️ Contains additionally for each of the 440 .min.js file a .min.js.gz file
    • 22 files differ:
      • ✅ Added version string in CSS as named in PR
        • e.g. media/com_installer/css/installer.min.css as ajax-loader.gif?v=144b8c
        • e.g. media/system/css/fields/calendar-rtl.min.css as select-bg-rtl.svg?v=e50af2
      • ⚠️ Removed first line @charset "UTF-8";
        • e.g. media/templates/administrator/atum/css/template.css
        • e.g. media/templates/site/cassiopeia/css/template-rtl.css
      • ✅ License is present
        • e.g. media/system/css/joomla-fontawesome.min.css
        • e.g. media/templates/site/cassiopeia/css/template.min.css
      • Have different content order
        • e.g. media/vendor/joomla.asset.json
  • Running npm run build:css
    • 1st time and saved a copy
      • ⚠️ This removes the 440 .min.js.gz files, the remaining media folder files are unchanged
    • ✅ 2nd time media folder files are unchanged
  • Running php build/build.php --exclude-gzip --exclude-zstd
    • Compared extracted media folder from built Joomla_5.3.4-rc1-Release_Candidate-Full_Package.zip with after-copy media folder
      • Differences are:
        • Different content order vendor/joomla.asset.json
        • .css files are missing new line and therefore .css.gz differ in size
        • ⚠️ version string is missing
          • e.g. media/com_installer/css/installer.min.css only ajax-loader.gif
          • e.g. media/plg_installer_webinstaller/css/client.min.css only ajax-loader.gif
        • ❌ File contains two lines @charset "UTF-8"; in the beginning
          • e.g. media/templates/administrator/atum/css/template.css
          • e.g. media/templates/site/cassiopeia/css/template-rtl.css
        • ✅ License is present now in media/system/css/joomla-fontawesome.min.css file
        • ❌ License is missing in media/templates/site/cassiopeia/css/template.min.css file

@muhme When using build.php you either have to use the —remote=HEAD parameter, or you have to create a new tag locally before running it, otherwise it will use the latest tag 5.3.4-rc1 for the build, which does not contain the changes from this PR.

@muhme When using build.php you either have to use the —remote=HEAD parameter, or you have to create a new tag locally before running it, otherwise it will use the latest tag 5.3.4-rc1 for the build, which does not contain the changes from this PR.

@richard67 Thank you, I'll retest later the day.

@muhme Another thing I should mention (also for other testers): The build.php script always uses either a tag or the head of the current branch and copies that into a new folder. Then composer and npm are run in that new folder. That means it will not use any changes which were made locally in the branch and have not been committed yet, except of changes on itself (script.php).

Thank you @muhme for testing. Some of your findings are not so clear to me, I hope it's okay if I ask a few more questions here.

Tested with JBT recreate 5.3.4-rc1

* Using PHP 8.4.7, Composer version 2.8.1, node v22.19.0 and npm 10.9.3

* Before the PR
  
  * Running `npm ci` and safed a before-copy of the `media` folder
  * Seen `media/vendor/bootstrap/css/bootstrap.min.css` contains the version info
  * Running `php build/build.php --exclude-gzip --exclude-zstd`
  * Unpacked media folder from built `Joomla_5.3.4-rc1-Release_Candidate-Full_Package.zip`
    
    * Seen `media/vendor/bootstrap/css/bootstrap.min.css` is missing the version info

* Applying the PR with `gh pr checkout 46146`
  
  * Running `npm ci` and safed a after-copy of the `media` folder
  * Compared `media` folders, the after PR changes are:
    
    * ⚠️ Contains additionally for each of the 440 `.min.js` file a `.min.js.gz` file

❓ I'm having trouble reproducing this. The gzip compression and the js files shouldn't actually be affected by this PR.
Could you please provide the complete path again so that I can better understand the issue?

    * 22 files differ:
      
      * ✅ Added version string in CSS as named in PR
        
        * e.g. `media/com_installer/css/installer.min.css` as `ajax-loader.gif?v=144b8c`
        * e.g. `media/system/css/fields/calendar-rtl.min.css` as `select-bg-rtl.svg?v=e50af2`
      * ⚠️ Removed first line `@charset "UTF-8";`

❓ Before the PR, I had the line @charset "UTF-8"; duplicated here, which should be fixed with this PR.
Has it been completely removed for you in these two files, or is it just the unwanted duplication that is no longer there?

        * e.g. `media/templates/administrator/atum/css/template.css`
        * e.g. `media/templates/site/cassiopeia/css/template-rtl.css`
      * ✅ License is present
        
        * e.g. `media/system/css/joomla-fontawesome.min.css`
        * e.g. `media/templates/site/cassiopeia/css/template.min.css`
      * Have different content order

❗ Thank you, get the same result. At the moment not completely clear to me here.
Some time needs to be invested here to determine whether this is intentional behaviour or not.

        * e.g. `media/vendor/joomla.asset.json`
  * Running `npm run build:css`
    
    * 1st time and saved a copy
      
      * ⚠️ This removes the 440 `.min.js.gz` files, the remaining `media` folder files are unchanged

❓ See above, unfortunately, I am unable to reproduce this at the moment.

    * ✅ 2nd time `media` folder files are unchanged

❓ I don't think I need to analyse this any further at this point. Thank you very much for the explanation, @richard67 .
#46146 (comment)

This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/46146.

It works, comments are there now.
However it is inconsistent with @charset "UTF-8"; for example it is in bootstrap.css but is missing in bootstrap-grid.css and in many un-minified files.

I have tested this item 🔴 unsuccessfully on 8b064e6

However it is inconsistent with @charset "UTF-8"; for example it is in bootstrap.css but is missing in bootstrap-grid.css and in many un-minified files.

But I think it is fine, the build script add @charset "UTF-8"; only to modified files, and the source files copying as it is.
Source of bootstrap.css contain @charset "UTF-8"; and source of bootstrap-grid.css does not.

The PR is good to me.

@brianteeman what issue did you get while testing?

Thanks @Fedik for the catch.
I just got the same result. We have an additional option --compile-bs for the bootstrap files.
We don't actually process them at all, and since they no longer go through the CSS versioning process in the end, they remain 1:1 like the original.

@Fedik Do you have any idea why the order in vendor\joomla.asset.json has changed?

I see more differences between the results without and with this PR than what was reported up to now, but they seem to fix issues. Unfortunately I'm busy at work. Will report more later tonight.

I have tested this item ✅ successfully on 8b064e6

Retested with local installation (w/o installing Joomla):

git clone https://github.com/joomla/joomla-cms 53
cd 53
gh pr checkout 46146
git tag test-46146 -m test-46146
composer i
npm ci

Saved media folder and rechecked first test findings:

• ✅ No additional .min.js.gz files
• ✅ Added version string in CSS as named in PR
  • e.g. media/com_installer/css/installer.min.css as ajax-loader.gif?v=144b8c
  • e.g. media/system/css/fields/calendar-rtl.min.css as select-bg-rtl.svg?v=e50af2
• ✅ First line @charset "UTF-8"; is existing:
  • e.g. media/templates/administrator/atum/css/template.css
  • e.g. media/templates/site/cassiopeia/css/template-rtl.css
• ✅ License is present now
  • e.g. media/system/css/joomla-fontawesome.min.css
  • e.g. media/templates/site/cassiopeia/css/template.min.css
• ✅ Running 1st time npm run build:css, no difference with saved media folder
• ✅ Running 2nd time npm run build:css, still no difference with saved media folder

Running php build/build.php --exclude-gzip --exclude-zstd --remote=test-46146
Rechecked first test findings in extracted media folder from built Joomla_5.3.4-rc2-dev-Development-Full_Package.zip
and in comparisation with saved media folder:

• Different content order vendor/joomla.asset.json
• ⚠️ Additional 440 .css.gz files only in full package media folder
• 36 files differ, all are .json with added version entry and sometimes changed order
• ✅ First line @charset "UTF-8"; is present
  • e.g. media/templates/administrator/atum/css/template.css
  • e.g. media/templates/site/cassiopeia/css/template-rtl.css
• ✅ License is present
  • e.g. media/system/css/joomla-fontawesome.min.css
  • e.g. media/templates/site/cassiopeia/css/template.min.css
• ✅ Version string is present
  • e.g. media/com_installer/css/installer.min.css for ajax-loader.gif
  • e.g. media/plg_installer_webinstaller/css/client.min.css for ajax-loader.gif
• ✅ File contains exactly one line @charset "UTF-8"; in the beginning
  • e.g. media/templates/administrator/atum/css/template.css
  • e.g. media/templates/site/cassiopeia/css/template-rtl.css

    ---

    _{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/46146.}

Do you have any idea why the order in vendor\joomla.asset.json has changed?

It modified in few places,
Once in localise-packages.mjs (while npm install) and second when running npm run versioning (while build.php is running). That can be result of it.

Differences found between build.php and npm ci

• media/system/css/system-site-general.min.css
  The hash for the URL of png images differs, e. g.
  • build: background:url(../images/j_button2_readmore.png?v=175861)
  • npm ci: background:url(../images/j_button2_readmore.png?v=175860)
    !! This image does not exists!
  • build: background:url(../images/j_button2_pagebreak.png?v=175861)
  • npm ci: background:url(../images/j_button2_pagebreak.png?v=175860)
    !! This image does not exist!
  • build: background:url(../images/j_button2_blank.png?v=175861)
  • npm ci: background:url(../images/j_button2_blank.png?v=175860)
    !! This image does not exist!
  • build: background:url(../images/selector-arrow.png?v=175861)
  • npm ci: background:url(../images/selector-arrow.png?v=175860)
    !! This image does not exist!

So here it would probably be appropriate to simply repair the styles. I imagine that this would fix the difference with this file.

The difference only in the hash v=xxx that is fine, because it is always new for each run.

I think in this case it is more a problem of - the images does not exist.
Found the same in

• media/templates/administrator/atum/css/vendor/minicolors/minicolors.min.css
  --> background-image:url(jquery.minicolors.png?v=175861)

I'm going to stop here now. There's no point in looking for an issue here when it's obviously lurking in the CSS files themselves.

Only for completeness - these two seem to have the same problem.
But at the moment, I don't have the time to analyse that.

• media/templates/administrator/atum/css/template-rtl.min.css
• media/templates/administrator/atum/css/template.min.css

I think in this case it is more a problem of - the images does not exist.

It looks like another issue, not related to current PR.

Additional 440 .css.gz files only in full package media folder

@muhme If you check more deeply you will see it is not only .css.gz but also .js.gz files. In general the gzipped resources are only created when building packages, not when just running npm.

@LadySolveig Here are my results. I've made 3 comparisons:

• "media" folder on a clean branch in a local git clone after composer install and npm ci without the PR and with the PR
• Full installation Zip package built without and with the PR
• Full installation Zip package built with the PR and "media" folder after composer install and npm ci with the PR

I'm not sure if my findings are ok or not.

Maybe @dgrammatiko and @Fedik could check and report back what they think?

Comparison of media folder with and without PR after composer installand npm ci

Minified CSS files with versions

With the PR, image URLs and font URLs have version appended in minified CSS files, while in the corresponding unminified files that's not the case.

Without the PR both minified and unminified don't have versions appended.

Is that right? And if not: Can we live with it?

If not: Maybe it can be fixed by not applying versioning on already minified files?

Files:

• com_installer/css/installer.min.css
• plg_installer_webinstaller/css/client.min.css
• system/css/joomla-fontawesome.min.css
• system/css/system-site-general.min.css
• templates/administrator/atum/css/template.min.css
• templates/administrator/atum/css/template-rtl.min.css
• templates/administrator/atum/css/vendor/choicesjs/choices.min.css
• templates/administrator/atum/css/vendor/fontawesome-free/fontawesome.min.css
• templates/administrator/atum/css/vendor/minicolors/minicolors.min.css
• templates/site/cassiopeia/css/template.min.css
• templates/site/cassiopeia/css/template-rtl.min.css
• templates/site/cassiopeia/css/global/fonts-local_roboto.min.css
• templates/site/cassiopeia/css/vendor/choicesjs/choices.min.css

Unminified CSS files with duplicate charset issue fixed

Duplicate @charset "UTF-8"; is fixed by the PR, no other changes in the file.

Files:

• templates/administrator/atum/css/template.css
• templates/administrator/atum/css/template-rtl.css
• templates/administrator/atum/css/vendor/choicesjs/choices.css
• templates/site/cassiopeia/css/template.css
• templates/site/cassiopeia/css/template-rtl.css
• templates/site/cassiopeia/css/vendor/choicesjs/choices.css

Minified template CSS files where Bootstrap copyright is added

The Bootstrap copyright is added because the files contain Bootstrap CSS at their top.

Is that right? And if not: Can we live with it?

The file have not been created by Bootstrap, they only partly consist of Bootstrap CSS.

Files:

• templates/administrator/atum/css/template.min.css
• templates/administrator/atum/css/template-rtl.min.css
• templates/site/cassiopeia/css/template.min.css
• templates/site/cassiopeia/css/template-rtl.min.css

4. Different odering of assets in joomla.asset.json files

Only one file affected: vendor/joomla.asset.json

Not a problem caused by this PR, happens also without it.

Comparison of full installation ZIP packages built with and without PR

Here all differences appear which are mentioned in the previous section and in the next section.

In addition, there are lots of differences in unminified CSS files which without the PR were modified e.g. by the linter, and now with the PR they are just taken as they are.

That also means that with the PR applied, these files don't get the @charset "UTF-8"; at the top.

This would make sense for vendor files which we don't want to modify.

But it is also the case for our files.

Example for media/com_associations/css/sidebyside.css, left hand side package built with the PR, right hand side package build without the PR:

There are lots of other cases for our component css, too many to list them here.

Comparison of full installation ZIP package with media folder, both with PR

Only the package contains the gzipped minified js and css files. This is as expected.

In all joomla.asset.json files ordering of assets differs, compared to the media folder of the local git clone after composer install and npm ci. This happens also without this PR.

"version": "xxxxxx" properties are added to assets in joomla.asset.json only in the package.
Is this right?
With PR #46147 that difference would not appear anymore, the assets would have versions in both cases.

There are no other differences between the media folder in the package and of the local git clone.

With the PR, image URLs and font URLs have version appended in minified CSS files, while in the corresponding unminified files that's not the case.

We do not process/modify existing source CSS files. Only minified files. #46146 (comment)

The Bootstrap copyright is added because the files contain Bootstrap CSS at their top.

The comment is presaved for each source is used when it containe one.
And because template uses some of scss files from boostrap the comment /*! ... */ from these files also presaved.

It is fine to me.

Different odering of assets in joomla.asset.json files

This because the build also run npm run versioning. #46146 (comment)

Should not be a problem. But yes, can be hard to compare the result.

Comparison of full installation ZIP packages built with and without PR

As meintoined we keep source CSS as it is. That probably the reason.
I would not worry about it, as long as minified version contain UTF-8.

In all joomla.asset.json files ordering of assets differs, compared to the media folder of the local git clone

The order does not come from filesystem, but from configuration json and script that processing these files.

It is fine to me.

"version": "xxxxxx" properties are added to assets in joomla.asset.json only in the package.
Is this right?

yes, it happen during build.php, npm run versioning command.

Different odering of assets in joomla.asset.json files

This because the build also run npm run versioning. #46146 (comment)

@Fedik That's not the only reason. The ordering seems to differ also between different runs of npm ci, or between different runs ob build.php.

I remember that we had more of these differences in past, and then a while (months, a year?) ago, some variables in the buils js scripts were changed from arrays to indexed lists or something like that, and then these differences became much less.

Maybe we can fix that, too? But not with this PR of course, because the copyright thing is urgent, and that thing not.

So for now we can live with that.

"version": "xxxxxx" properties are added to assets in joomla.asset.json only in the package.
Is this right?
yes, it happen during build.php, npm run versioning command.

With the other PR #46147 that would be the same for both cases, always version properties added.

Assuming that we want to use local branches for testing, and it makes sense to test what end users will see, I currently would prefer the other PR, which besides that point has the same results as this one here.

But as the license issue is the most urgent, and this point is just nice to have, we can use this PR here for the upcoming releases, and the other one we can use later at any time as that has only an effect on a development environment but not for the end user.

The ordering seems to differ also between different runs of npm ci

Hm, then need to look each particular case. Example, for codemirror the script generate new elements while npm build (while npm install), the order may depend from "node modules" lookup script.

But yes, not for this PR.

I have tested this item 🔴 unsuccessfully on 8b064e6

@brianteeman Could you report back what has failed for you? Was it something from my findings? Or was the copyright comment missing somewhere, and I have missed that in my findings?

@dgrammatiko What do you think about my findings above? Can we live with that?

Comparison of media folder with and without PR after composer installand npm ci

Minified CSS files with versions

With the PR, image URLs and font URLs have version appended in minified CSS files, while in the corresponding unminified files that's not the case.

Without the PR both minified and unminified don't have versions appended.

Is that right? And if not: Can we live with it?

If not: Maybe it can be fixed by not applying versioning on already minified files?

Than you would have exactly the same problem as before, that we try to solve.
The step for minifaction and and versioning has now been combined into a single call.
Previously, we made the call once with npm ci and then went through all the files again and basically did the whole thing again with the additional plugin for versioning.
And on the second call, the copyright information was removed because it was no longer in the first line.

Minified template CSS files where Bootstrap copyright is added

The Bootstrap copyright is added because the files contain Bootstrap CSS at their top.

Is that right? And if not: Can we live with it?

The file have not been created by Bootstrap, they only partly consist of Bootstrap CSS.

To be honest, I can't think of a way to remove that at the moment. Until now, everything has been removed as an accident.
If you want to decide on a case-by-case basis, I think you would have to do a lot manually.

I think it would also be important to mention in this context that we should adapt our own copyright headers so that they are not removed either.
At them moment the ! ist missing, (/*! ... */)

From what I've seen, we're not really consistent here either. I just don't know if it's intentional that we sometimes have one and often don't, such as in the Cassiopeia template, where we don't have a copyright at all.

Comparison of full installation ZIP packages built with and without PR

Here all differences appear which are mentioned in the previous section and in the next section.

In addition, there are lots of differences in unminified CSS files which without the PR were modified e.g. by the linter, and now with the PR they are just taken as they are.

That also means that with the PR applied, these files don't get the @charset "UTF-8"; at the top.

This would make sense for vendor files which we don't want to modify.

But it is also the case for our files.

If I understand correctly, the long-term goal is to have a mostly comparable result in the media files for dev and for the build packages.

But then it is actually the case that, especially with the original CSS files, it makes the most sense to pay attention to the correct format directly and, for example, to have the @charset ‘UTF-8’; at the top directly in the file.

SCSS files always have to be compiled, so it's absolutely no problem to process such steps, and the compiled files don't go back into the Git repository after calling npm ci.

But any changes made by the linter to the CSS files will always remain different between build / dev files and the file we store via git.
Worse still, it would be marked as a changed file every time in the developer branch if you would combine that with npm ci.
~~So ultimately, we have to live with either ~~
~~copying the css files 1:1 (then they are comparable) ~~
or accepting that they are different in the build package.

Of course, it would be best if they were optimised before being added to the repository.
That's what we do with the php files, after all.

That was nonsense, sorry. As we don't do it in place.

With the other PR #46147 that would be the same for both cases, always version properties added.

Assuming that we want to use local branches for testing, and it makes sense to test what end users will see, I currently would prefer the other PR, which besides that point has the same results as this one here.

The longer I think about this one, the more I think we should perhaps test it in 6.1, if we really want to, as we will have more time until the productive build and it will probably be called up and tested more frequently.

Of course, it would be best if they were optimised before being added to the repository.
That's what we do with the php files, after all.

Thats the obvious example that we have a problem with the build scripts. lint:css and lint:js run on every PR so there never should be linting changes in a build

So there is absolutely no reason that the build script generates something different. The only explanation is that the build scripts are NOT configured the same as npm ci

There is no reason ever for the output of the build script to be different to npm ci EXCEPT for a difference in the version number.

How can we ever say that we test something if the build scripts produces different output. Stripping the copyright statements AND version numbers [1] was just the straw that broke the camels back and forced us to look at all the files and see how badly the build scripts are broken

[1] a user did report this but I (no one else bothered to respond) disputed the user report as I couldnt believe that we were doing something so daft as to ship code that was in any way different to the tested code

I have tested this item 🔴 unsuccessfully on 8b064e6

@brianteeman Could you report back what has failed for you? Was it something from my findings? Or was the copyright comment missing somewhere, and I have missed that in my findings?

I said I was offline but as I saw such misinformation and bad practice in this thread I broke my break - i am returning to offline mode now

The code style check runs in the build folder and the raw source files and has, the way it is carried out, nothing to do with what is compiled by the npm ci call or the build script.
The code style check runs on the raw source files and not the compiled ones.

There is no reason ever for the output of the build script to be different to npm ci EXCEPT for a difference in the version number.

@brianteeman Then you should support this PR because it reduces these differences. Without this PR e.g. the "media/com_installer/css/installer.css" differs between the result of composer install and npm ci on a local branch, with this PR both are the same. Same for other files.

There is no reason ever for the output of the build script to be different to npm ci EXCEPT for a difference in the version number.

@brianteeman Then you should support this PR because it reduces these differences. Without this PR e.g. the "media/com_installer/css/installer.css" differs between the result of composer install and npm ci on a local branch, with this PR both are the same. Same for other files.

No I should do what I believe not what you decide I should believe

There is no reason ever for the output of the build script to be different to npm ci EXCEPT for a difference in the version number.

@brianteeman Then you should support this PR because it reduces these differences. Without this PR e.g. the "media/com_installer/css/installer.css" differs between the result of composer install and npm ci on a local branch, with this PR both are the same. Same for other files.

No I should do what I believe not what you decide I should believe

Please do me a favour and be destructive somewhere else. This is complex enough as it is.
If you want to help, I'd be grateful.
But demotivation is the last thing I need right now.

There was no need for that comment

I have now added an additional fallback check for the vendor files so that we don't lose any copyright, even if it is not available in the format required by lightningcss.

All files, both our own and the vendor CSS files, are now being processed again.
Unfortunately, directly copying the files often resulted in the leading line for UTF-8 being missing, especially in the vendor files. The result is also closer to our previous handling.

I have tested this item ✅ successfully on fb5ccaa

Retested with the following commands, which were noted down to document how the tests were performed and to provide the opportunity to repeat or continue the tests.

    git clone https://github.com/joomla/joomla-cms 53
    cd 53
    gh pr checkout 46146
    git tag test-46146 -m test-46146
    composer i
    npm ci
    cp -pr media /tmp/media_first
    php build/build.php --exclude-gzip --exclude-zstd --remote=test-46146
    unzip build/tmp/packages/Joomla_5.3.4-rc2-dev-Development-Full_Package.zip "media/*" -d /tmp
    mv /tmp/media /tmp/media_package

• ✅ Test 1 Check copyright information in the media files
  • media/vendor/bootstrap/css/bootstrap.css
  • media/vendor/bootstrap/css/bootstrap.min.css
  • media/templates/administrator/atum/css/vendor/fontawesome-free/fontawesome.css
  • media/templates/administrator/atum/css/vendor/fontawesome-free/fontawesome.min.css
• ✅ Test 2 Check the media files also in the package, that is build from this PR
  • Simple compared all 217 .css files, they are not only existing, they are are equal:

    cd /tmp/media_first; find . -name \*.css | while read f; do cmp $f /tmp/media_package/$f; done
    

• ✅ Test 3 Check that @charset "UTF-8"; is on top of each css file and at the beginning of the min.css files
  • Checked the 217 files with:

    find /tmp/media_first -name \*.css | xargs grep -aL '^@charset "UTF-8";' --
    find /tmp/media_package -name \*.css | xargs grep -aL '^@charset "UTF-8";' --
  

• ✅ Test 4 Check that the site is working as expected and no media files are missing.
  • Installed manually on localhost from extracted full package
  • Enabled 'Debug System' and 'Log Almost Everything'
  • Created article, viewed and edited on the frontend
  • Uploaded image, cropped and rotated
  • Sended test mail
  • Added Smart Search and tested on frontend
  • Checked Joomla logs, PHP log and HTTP access log for ' 404 '
• ⚠️ Test 5 Compare the media files from build package with npm ci
  • As described:

    rm -rf media /tmp/media_*
    php ./build/build.php --remote=HEAD
    unzip build/tmp/packages/Joomla_5.3.4-rc2-dev-Development-Full_Package.zip "media/*" -d /tmp
    mv /tmp/media /tmp/media_package
    npm ci
    cp -r media /tmp/media_npm
  

  • /tmp/media_first contains 1'661 files, /tmp/media_package contains 2'101 files
    • Difference are additional .gz files in package, deleted with:
      diff -r /tmp/media_npm /tmp/media_package |grep 'Only in ' | sed -e 's/Only in //' -e 's|: |/|' | xargs rm
  • Remaining differences are checked with diff -r /tmp/media_npm /tmp/media_package|more:
    • 36 files differ as seen with diff -rq /tmp/media_npm /tmp/media_package | wc -l
    • All 36 files are *.joomla.asset.json
    • Only one file (vendor/joomla.asset.json) is equal in size

      diff -rq . ../media_npm | sed -e 's|Files ./||' -e 's/ and .*//' | while read f ; do [ $(wc -c < $f) -ne $(wc -c < ../media_npm/$f) ] || echo "$f equal" ; done
      

    • Differences are attached as:

      diff -r /tmp/media_npm /tmp/media_package > /tmp/diff.txt
    

    • ⚠️ Please check diff.txt

Unfortunately, directly copying the files often resulted in the leading line for UTF-8 being missing, especially in the vendor files. The result is also closer to our previous handling.

Do not worry about them, the source files only used in debug mode, kind of for "light debugging", can ignore them.
However in future we could address them also.

However, since, at least in my understanding, the primary purpose of the hashes is that they should change when there is a real change in content of the files, I have now taken this approach.

It is not strict. The purpuse to force cache refresh in User browser.
Dimitris coded it to be from file hash for JS and random hash for CSS url().

For dev build it can be the build time or fixed hash somewhere in config.
For the final/release build it could be the build version (Actually, for Dev also, in theory).

No need to invent anything to fancy here.

But again, this PR is already good as it is: fixing copyright.

The "files difference" can be addressed somewhere else if someone really want,
Personally I do not see it as a big problem and would not spend much time on it.

@muhme The difference in size of the joomla.asset.json files when comparing the npm ci result with the package comes from the version properties for assets being added only for the package. That is ok from my point of view.

The PR does not only fix the copyright issue and duplicate UTF-8 thing issue.

It fixes also an ugly issue for the installation template CSS files (installation/template/css/template.css and installation/template/css/template-rtl.css and their minified files) which I've just noticed for the first time:

The fontawesome vendor stuff is included two times without this PR in the mentioned files in the packages.

This is fixed by this PR.

Will continue my checks and post results later, but up to now it looks good. Differences between media folder on the git clone and in the package are reduced much. License issues are solved.

One more thing about versioning hash. In theory it possible to update WebAssetmanager to read the main version from json and append to children assets if their version not exists. Then in build we can update only main version in the json.

I have this idea for few years, but to lazy to implement it 😄

The PR does not only fix the copyright issue and duplicate UTF-8 thing issue.

Yes, you are right. I think most critical was copyright.

@muhme I've just checked your diff.txt. It fits to my findings. The versions are only added in the package, and the ordering of assets differ (know issue to be fixed with a future PR).

I have tested this item ✅ successfully on fb5ccaa

The PR fixes the copyright issue.

In addition, it fixes an issue with the installation template files. Without this PR they have duplicate CSS due to multiple inclusion of the fontawesome vendor CSS and a duplicate "@charset" line in the built packages, with the PR this is fixed.

Differences between the media folder on the local git clone after a "composer install" and an "npm ci" and the media folder in the packages built with the build.php script are drastically reduced. The only differences with the PR are:

• Gzipped minified js and css files exist only in the packages. That's as expected.
• In the joomla.asset.json files, version properties are added to assets only in the packages. That's also as expected.
• Ordering of assets still differs in some joomla.asset.json files, but that is also the case without this PR.

What I've also checked: After 2 consecutive runs of npm ci, the only difference is a slightly different order of assets in the one file vendor\joomla.asset.json. Everything else is the same.

Finally I've checked that the fix for the installation template CSS mentioned before has no visual impact. I've mane a new installation with the package created by the build.php script with this PR applied, and have checked that everything works and looks as usual and there are no errors in the browser console.

So besides fixing the mentioned issues, this PR is a huge step into direction for 100% reproducible build results.

Amazing work, thanks a lot @LadySolveig .

RTC

I have tested this item ✅ successfully on fb5ccaa

I checked about 40 files by hand and all of them have now the correct copyright header and Utf8 statement. Json ordering and different css order after compression is a bit annoying... but that's for another pr.

Thank you all very much for your fantastic support and cooperation. I am very grateful to you for the extensive testing!