? Language Change NPM Resource Changed PR-4.3-dev Pending

User tests: Successful: Unsuccessful:

avatar obuisard
obuisard
3 Jan 2023
avatar joomla-cms-bot joomla-cms-bot - change - 3 Jan 2023
Category Unit Tests Administration SQL com_admin Postgresql com_categories com_content com_contenthistory com_fields com_finder JavaScript com_media NPM Change com_templates com_users Language & Strings Modules Repository Front End
avatar obuisard obuisard - open - 3 Jan 2023
avatar obuisard obuisard - change - 3 Jan 2023
Status New Pending
avatar joomla-cms-bot
joomla-cms-bot - comment - 3 Jan 2023

Please add more information to your issue. Without test instructions and/or any description we will close this issue within 4 weeks. Thanks.
This is an automated message from the J!Tracker Application.

avatar obuisard obuisard - change - 3 Jan 2023
Title
Upmerge 2023 01 02
Updating 4.3-dev with changes from 4.2-dev
avatar obuisard obuisard - edited - 3 Jan 2023
avatar obuisard
obuisard - comment - 3 Jan 2023

I have spent hours fixing conflicts in package-lock.json.
I did it to the best of my ability in trying to understand the changes, what needed to be changed and what needed to remain in 4.3.
However, I would really appreciate someone checking the file out to ensure I did not make any mistake. There is no way there are no mistakes in the file over the >50 conflicts that needed to be looked over.

Thank you for your help.
@HLeithner @wilsonge

avatar wilsonge
wilsonge - comment - 3 Jan 2023

Drone failed because package-lock isn't representing what's in the package.json. My best advice is don't care. This branch isn't out yet and whilst it's not ideal to keep bumping dependencies through beta - the easiest way to fix this is to just run an npm i and grab the latest dependencies and be done with it. It's nearly impossible to keep those in sync like you say...

avatar HLeithner
HLeithner - comment - 3 Jan 2023

as george said, you have to ignore the package-lock.json and the composer.lock just rebuild it if they conflict.

avatar obuisard obuisard - change - 3 Jan 2023
Labels Added: ? ? Language Change NPM Resource Changed PR-4.3-dev
avatar dgrammatiko
dgrammatiko - comment - 3 Jan 2023

@obuisard delete the lock files, rerun composer install and npm install and commit then the lock files. FWIW the ranges in the jsons is wrong for this project, the versions on these files SHOULD represent exactly the version installed!!!

avatar obuisard
obuisard - comment - 4 Jan 2023

Thank you all for your help and advice. Traveling right now so I will hold this until Friday at the latest for such procedure that requires full attention :-)

avatar brianteeman
brianteeman - comment - 6 Jan 2023

Maybe this was the wrong way to do it but this is what I did and it failed :(

  1. apply PR
  2. delete the two lcok files
  3. comppser install
  4. npm install

It failed on step 4 with the following error which I will happily admit to failing to understand


179 verbose stack Error: command failed
179 verbose stack     at ChildProcess.<anonymous> (C:\Users\brian\AppData\Roaming\npm\node_modules\npm\node_modules\@npmcli\promise-spawn\lib\index.js:63:27)
179 verbose stack     at ChildProcess.emit (node:events:513:28)
179 verbose stack     at maybeClose (node:internal/child_process:1098:16)
179 verbose stack     at ChildProcess._handle.onexit (node:internal/child_process:304:5)
180 verbose pkgid joomla@4.0.0
181 verbose cwd C:\htdocs\j42
182 verbose Windows_NT 10.0.22623
183 verbose node v19.1.0
184 verbose npm  v8.19.2
185 error code 1
186 error path C:\htdocs\j42
187 error command failed
188 error command C:\WINDOWS\system32\cmd.exe /d /s /c node build/build.js --prepare
189 verbose exit 1
190 timing npm Completed in 114090ms
191 verbose code 1
avatar brianteeman
brianteeman - comment - 6 Jan 2023

I see now that drone reported the exact same error

avatar dgrammatiko
dgrammatiko - comment - 6 Jan 2023

@obuisard @brianteeman my bad I guess, update the scripts part of the package.json to:

    "build:js": "node build/build.js --compile-js",
    "build:css": "node build/build.js --compile-css",
    "build:bs5": "node build/build.js --compile-bs",
    "build:com_media": "set NODE_ENV=PRODUCTION && node build/build.js --com-media",
    "build:com_media:dev": "set NODE_ENV=DEVELOPMENT && node build/build.js --com-media",
    "watch": "node build/build.js --watch",
    "watch:com_media": "node build/build.js --watch-com-media",
    "lint:js": "eslint --config build/.eslintrc --ignore-pattern '/media/' --ext .es6.js,.es6,.vue .",
    "lint:css": "stylelint --config build/.stylelintrc.json \"administrator/components/com_media/resources/**/*.scss\" \"administrator/templates/**/*.scss\" \"build/media_source/**/*.scss\" \"templates/**/*.scss\" \"installation/template/**/*.scss\"",
    "install": "set NODE_ENV=PRODUCTION && node build/build.js --prepare",
    "update": "set NODE_ENV=PRODUCTION && node build/build.js --copy-assets && node build/build.js --build-pages && node build/build.js --compile-js && node build/build.js --compile-css && node build/build.js --compile-bs && node build/build.js --com-media",
    "gzip": "node build/build.js --gzip",
    "versioning": "node build/build.js --versioning",
    "browserlist:update": "npx browserslist@latest --update-db",
    "cypress:install": "cypress install",
    "cypress:open": "cypress open",
    "cypress:run": "cypress run"

There are couple of missing env variables...

avatar brianteeman
brianteeman - comment - 6 Jan 2023

dam - just came back to report that it was all your fault as there were missing variables

/me wonders if anyojne tested the pr

avatar brianteeman
brianteeman - comment - 6 Jan 2023

@dgrammatiko your change didnt work. it seems to be a windows issue that needs different sytax when using node_env than you have used.

https://stackoverflow.com/questions/9249830/how-can-i-set-node-env-production-on-windows

avatar dgrammatiko
dgrammatiko - comment - 6 Jan 2023

/me wonders if anyojne tested the pr

They did and actually the PR without the extra ENV vars is working perfectly on the 4.2 repo. So basically this is the best proof that ranges both in package.json and composer.json are extremely dangerous as you end up with versions that might actually break the system. I keep repeating, that both these sons should point to SPECIFIC versions of the packages but I guess nobody really understands what I'm talking about. Maybe this could spark some discussion in the maintainers group, I mean nothing really changes since there's the renovate bot that updates things whenever a newer version is available (the bot's configuration allows for ranges per package, etc, but the json have specific versions!!!)

avatar brianteeman
brianteeman - comment - 6 Jan 2023

/me l;eaves this to cleverer people than me. am in over my head now

avatar dgrammatiko
dgrammatiko - comment - 6 Jan 2023

it seems to be a windows issue that needs different sytax when using node_env than you have used.

can you confirm that set NODE_ENV=PRODUCTION works? (I can't test it, not on windows)

EDIT I've updated the script section on the comment above to fit windows OS

avatar brianteeman
brianteeman - comment - 6 Jan 2023

still no luck

avatar brianteeman
brianteeman - comment - 6 Jan 2023

just to confirm that npm install with 4.2 completes successfuly

avatar obuisard
obuisard - comment - 6 Jan 2023

Thank you, Brian @brianteeman for testing this and @dgrammatiko for looking into it.
I will add the changes you submitted and try it as well. I am on Windows but use WSL with Debian.

avatar dgrammatiko
dgrammatiko - comment - 6 Jan 2023

@obuisard could your try this:

    "build:js": "node build/build.js --compile-js",
    "build:css": "node build/build.js --compile-css",
    "build:bs5": "node build/build.js --compile-bs",
    "build:com_media": "set NODE_ENV=PRODUCTION && node build/build.js --com-media",
    "build:com_media:dev": "set NODE_ENV=DEVELOPMENT && node build/build.js --com-media",
    "watch": "node build/build.js --watch",
    "watch:com_media": "node build/build.js --watch-com-media",
    "lint:js": "eslint --config build/.eslintrc --ignore-pattern '/media/' --ext .es6.js,.es6,.vue .",
    "lint:css": "stylelint --config build/.stylelintrc.json \"administrator/components/com_media/resources/**/*.scss\" \"administrator/templates/**/*.scss\" \"build/media_source/**/*.scss\" \"templates/**/*.scss\" \"installation/template/**/*.scss\"",
    "install": "node build/build.js --prepare",
    "update": "node build/build.js --copy-assets && node build/build.js --build-pages && node build/build.js --compile-js && node build/build.js --compile-css && node build/build.js --compile-bs && set NODE_ENV=PRODUCTION && node build/build.js --com-media",
    "gzip": "node build/build.js --gzip",
    "versioning": "node build/build.js --versioning",
    "browserlist:update": "npx browserslist@latest --update-db",
    "cypress:install": "cypress install",
    "cypress:open": "cypress open",
    "cypress:run": "cypress run"
avatar dgrammatiko
dgrammatiko - comment - 6 Jan 2023

@obuisard one more try, add this at the end of the package.json

  "optionalDependencies": {
     "sass-embedded-linux-x64": "^1.57.1"
   }

In case it's the same issue I had here: #39400

avatar brianteeman
brianteeman - comment - 7 Jan 2023

tested again.

I think the problem occurs because the file media\com_media\js\media-manager.js is not being created first

image

avatar dgrammatiko
dgrammatiko - comment - 7 Jan 2023

@brianteeman could you try changing this:

if (isProduction) {
// eslint-disable-next-line no-console
console.log('✅ ES2017 Media Manager ready');
minifyJs('media/com_media/js/media-manager.js');
return buildLegacy(resolve('media/com_media/js/media-manager.js'));
}
// eslint-disable-next-line no-console
console.log('✅ ES2017 Media Manager ready');
copyFile('media/com_media/js/media-manager.js', 'media/com_media/js/media-manager.js');
return '';
};

to:

 if (isProduction) {
    // eslint-disable-next-line no-console
    console.log('✅ ES2017 Media Manager ready');
    minifyJs('media/com_media/js/media-manager.js');
    return buildLegacy(resolve('media/com_media/js/media-manager.js'));
  } else {
    // eslint-disable-next-line no-console
    console.log('✅ ES2017 Media Manager ready');
    copyFile('media/com_media/js/media-manager.js', 'media/com_media/js/media-manager.js');
    return '';
}

Although I think there's problem with either rollup (or some plugin) or SCSS or postcss, as the MM is the last step and the error is immediately after the localisation of the node packages

avatar brianteeman
brianteeman - comment - 7 Jan 2023
  1. same error
  2. the file you linked to was the version for 4.2 - the version in this branch is different
avatar brianteeman
brianteeman - comment - 7 Jan 2023

and as I said these lines cant be the problem because they are trying to minify a file that does not exist (yet)

avatar dgrammatiko
dgrammatiko - comment - 7 Jan 2023

@brianteeman good point, @obuisard please replace the contents of build/build-modules-js/javascript/build-com_media-js.es6.js with (the contents of the 4.2 respective file):

const { resolve } = require('path');
const { copyFile } = require('fs').promises;
const rollup = require('rollup');
const { nodeResolve } = require('@rollup/plugin-node-resolve');
const replace = require('@rollup/plugin-replace');
const { babel } = require('@rollup/plugin-babel');
const VuePlugin = require('rollup-plugin-vue');
const commonjs = require('@rollup/plugin-commonjs');
const { minifyJs } = require('./minify.es6.js');
require('dotenv').config();

const inputJS = 'administrator/components/com_media/resources/scripts/mediamanager.es6.js';
const isProduction = process.env.NODE_ENV !== 'DEVELOPMENT';

const buildLegacy = async (file) => {
  // eslint-disable-next-line no-console
  console.log('Building Legacy Media Manager...');

  const bundle = await rollup.rollup({
    input: file,
    plugins: [
      nodeResolve(),
      commonjs(),
      babel({
        exclude: 'node_modules/core-js/**',
        babelHelpers: 'bundled',
        babelrc: false,
        presets: [
          [
            '@babel/preset-env',
            {
              corejs: '3.8',
              useBuiltIns: 'usage',
              targets: {
                ie: '11',
              },
              loose: true,
              bugfixes: true,
              modules: false,
              ignoreBrowserslistConfig: true,
            },
          ],
        ],
      }),
    ],
    external: [],
  });

  await bundle.write({
    format: 'iife',
    sourcemap: false,
    name: 'JoomlaMediaManager',
    file: 'media/com_media/js/media-manager-es5.js',
  });

  // closes the bundle
  await bundle.close();

  // eslint-disable-next-line no-console
  console.log('Legacy Media Manager ready ✅');

  minifyJs('media/com_media/js/media-manager-es5.js');
};

module.exports.mediaManager = async () => {
  // eslint-disable-next-line no-console
  console.log('Building Media Manager ES Module...');

  const bundle = await rollup.rollup({
    input: resolve(inputJS),
    plugins: [
      VuePlugin({
        target: 'browser',
        css: false,
        compileTemplate: true,
        template: {
          isProduction,
        },
      }),
      nodeResolve(),
      commonjs(),
      replace({
        'process.env.NODE_ENV': JSON.stringify((process.env.NODE_ENV && process.env.NODE_ENV.toLocaleLowerCase()) || 'production'),
        __VUE_OPTIONS_API__: true,
        __VUE_PROD_DEVTOOLS__: isProduction,
        preventAssignment: true,
      }),
      babel({
        exclude: 'node_modules/core-js/**',
        babelHelpers: 'bundled',
        babelrc: false,
        presets: [
          [
            '@babel/preset-env',
            {
              targets: {
                browsers: [
                  '> 1%',
                  'not ie 11',
                  'not op_mini all',
                ],
              },
              loose: true,
              bugfixes: false,
              ignoreBrowserslistConfig: true,
            },
          ],
        ],
      }),
    ],
  });

  await bundle.write({
    format: 'es',
    sourcemap: false,
    file: 'media/com_media/js/media-manager.js',
  });

  // closes the bundle
  await bundle.close();

  if (isProduction) {
    // eslint-disable-next-line no-console
    console.log('✅ ES2017 Media Manager ready');
    minifyJs('media/com_media/js/media-manager.js');
    return buildLegacy(resolve('media/com_media/js/media-manager.js'));
  }
  // eslint-disable-next-line no-console
  console.log('✅ ES2017 Media Manager ready');
  copyFile('media/com_media/js/media-manager.js', 'media/com_media/js/media-manager.js');
  return '';
};

module.exports.watchMediaManager = async () => {
  // eslint-disable-next-line no-console
  console.log('Watching Media Manager js+vue files...');
  // eslint-disable-next-line no-console
  console.log('=========');

  const watcher = rollup.watch({
    input: resolve(inputJS),
    plugins: [
      VuePlugin({
        target: 'browser',
        css: false,
        compileTemplate: true,
        template: {
          isProduction: true,
        },
      }),
      nodeResolve(),
      replace({
        preventAssignment: true,
        'process.env.NODE_ENV': JSON.stringify('production'),
      }),
      babel({
        exclude: 'node_modules/core-js/**',
        babelHelpers: 'bundled',
        babelrc: false,
        presets: [
          [
            '@babel/preset-env',
            {
              targets: {
                browsers: [
                  '> 1%',
                  'not ie 11',
                  'not op_mini all',
                ],
              },
              loose: true,
              bugfixes: false,
              ignoreBrowserslistConfig: true,
            },
          ],
        ],
      }),
    ],
    output: [
      {
        format: 'es',
        sourcemap: false,
        file: 'media/com_media/js/media-manager.js',
      },
      {
        format: 'es',
        sourcemap: false,
        file: 'media/com_media/js/media-manager.min.js',
      },
    ],
  });

  watcher.on('event', (event) => {
    if (event.code === 'BUNDLE_END') {
      // eslint-disable-next-line no-console
      console.log(`File ${event.output[0]} updated ✅
File ${event.output[1]} updated ✅
=========`);
    }
  });
};
avatar brianteeman
brianteeman - comment - 7 Jan 2023

tested with the replacement file

  1. npm install ✔️
  2. npm run build:com_media ✔️
  3. npm run build:com_media:dev✔️ (not sure what it does but no errors)
avatar dgrammatiko
dgrammatiko - comment - 7 Jan 2023

(not sure what it does but no errors)

  • it doesn't create the -es5.js...
  • the minified file is not minified (it's a copy of the .js)
avatar obuisard
obuisard - comment - 7 Jan 2023

Thanks Dimitris and Brian for trying to get to the bottom of this. Unfortunately, the build is still failing.

avatar dgrammatiko
dgrammatiko - comment - 7 Jan 2023

@obuisard it seems that 4.2 and 4.3 have different build tools, try replacing the contents with

build/build-modules-js/javascript/build-com_media-js.es6.js
const { resolve } = require('path');
const { writeFile } = require('fs').promises;
const rollup = require('rollup');
const { nodeResolve } = require('@rollup/plugin-node-resolve');
const replace = require('@rollup/plugin-replace');
const { babel } = require('@rollup/plugin-babel');
const VuePlugin = require('rollup-plugin-vue');
const commonjs = require('@rollup/plugin-commonjs');
const { minifyJsCode } = require('./minify.es6.js');

const inputJS = 'administrator/components/com_media/resources/scripts/mediamanager.es6.js';
const isProduction = process.env.NODE_ENV !== 'DEVELOPMENT';

const buildLegacy = async (file) => {
  // eslint-disable-next-line no-console
  console.log('Building Legacy Media Manager...');

  const bundle = await rollup.rollup({
    input: file,
    plugins: [
      nodeResolve(),
      commonjs(),
      babel({
        exclude: 'node_modules/core-js/**',
        babelHelpers: 'bundled',
        babelrc: false,
        presets: [
          [
            '@babel/preset-env',
            {
              corejs: '3.8',
              useBuiltIns: 'usage',
              targets: {
                ie: '11',
              },
              loose: true,
              bugfixes: true,
              modules: false,
              ignoreBrowserslistConfig: true,
            },
          ],
        ],
      }),
    ],
    external: [],
  });

  await bundle.write({
    format: 'iife',
    sourcemap: false,
    name: 'JoomlaMediaManager',
    file: 'media/com_media/js/media-manager-es5.js',
  })
    .then((value) => minifyJsCode(value.output[0].code))
    .then((content) => {
      // eslint-disable-next-line no-console
      console.log('✅ Legacy Media Manager ready');
      return writeFile(resolve('media/com_media/js/media-manager-es5.min.js'), content.code, { encoding: 'utf8', mode: 0o644 });
    })
    .catch((error) => {
      // eslint-disable-next-line no-console
      console.error(error);
    });

  // closes the bundle
  await bundle.close();
};

module.exports.mediaManager = async () => {
  // eslint-disable-next-line no-console
  console.log('Building Media Manager ES Module...');

  const bundle = await rollup.rollup({
    input: resolve(inputJS),
    plugins: [
      VuePlugin({
        target: 'browser',
        css: false,
        compileTemplate: true,
        template: {
          isProduction,
        },
      }),
      nodeResolve(),
      commonjs(),
      replace({
        'process.env.NODE_ENV': JSON.stringify((process.env.NODE_ENV && process.env.NODE_ENV.toLocaleLowerCase()) || 'production'),
        __VUE_OPTIONS_API__: true,
        __VUE_PROD_DEVTOOLS__: isProduction,
        preventAssignment: true,
      }),
      babel({
        exclude: 'node_modules/core-js/**',
        babelHelpers: 'bundled',
        babelrc: false,
        presets: [
          [
            '@babel/preset-env',
            {
              targets: {
                browsers: [
                  '> 1%',
                  'not ie 11',
                  'not op_mini all',
                ],
              },
              loose: true,
              bugfixes: false,
              ignoreBrowserslistConfig: true,
            },
          ],
        ],
      }),
    ],
  });

  bundle.write({
    format: 'es',
    sourcemap: !isProduction,
    file: 'media/com_media/js/media-manager.js',
  })
    .then((value) => minifyJsCode(value.output[0].code))
    .then((content) => {
      if (isProduction) {
        // eslint-disable-next-line no-console
        console.log('✅ ES2017 Media Manager ready');
        writeFile(resolve('media/com_media/js/media-manager.min.js'), content.code, { encoding: 'utf8', mode: 0o644 });
        return buildLegacy(resolve('media/com_media/js/media-manager.js'));
      } else {
        // eslint-disable-next-line no-console
        console.log('✅ ES2017 Media Manager ready');
        writeFile(resolve('media/com_media/js/media-manager.js'), content.code, { encoding: 'utf8', mode: 0o644 });
        return;
      }
    })
    .then(() => buildLegacy(resolve('media/com_media/js/media-manager.js')))
    .catch((error) => {
      // eslint-disable-next-line no-console
      console.error(error);
    });

  // closes the bundle
  await bundle.close();
};

module.exports.watchMediaManager = async () => {
  // eslint-disable-next-line no-console
  console.log('Watching Media Manager js+vue files...');
  // eslint-disable-next-line no-console
  console.log('=========');

  const watcher = rollup.watch({
    input: resolve(inputJS),
    plugins: [
      VuePlugin({
        target: 'browser',
        css: false,
        compileTemplate: true,
        template: {
          isProduction: true,
        },
      }),
      nodeResolve(),
      replace({
        preventAssignment: true,
        'process.env.NODE_ENV': JSON.stringify('production'),
      }),
      babel({
        exclude: 'node_modules/core-js/**',
        babelHelpers: 'bundled',
        babelrc: false,
        presets: [
          [
            '@babel/preset-env',
            {
              targets: {
                browsers: [
                  '> 1%',
                  'not ie 11',
                  'not op_mini all',
                ],
              },
              loose: true,
              bugfixes: false,
              ignoreBrowserslistConfig: true,
            },
          ],
        ],
      }),
    ],
    output: [
      {
        format: 'es',
        sourcemap: false,
        file: 'media/com_media/js/media-manager.js',
      },
      {
        format: 'es',
        sourcemap: false,
        file: 'media/com_media/js/media-manager.min.js',
      },
    ],
  });

  watcher.on('event', (event) => {
    if (event.code === 'BUNDLE_END') {
      // eslint-disable-next-line no-console
      console.log(`✅ File ${event.output[0]} updated
✅ File ${event.output[1]} updated
=========`);
    }
  });
};
avatar obuisard
obuisard - comment - 8 Jan 2023

No matter what, we get:

Processing Vendor file: media/vendor/webcomponentsjs/js/webcomponents-bundle.js
Error: Can't call SyncProcess.yield() after the process has exited.
npm ERR! code 1
npm ERR! path /********/src
npm ERR! command failed
npm ERR! command sh -c -- node build/build.js --prepare
avatar dgrammatiko
dgrammatiko - comment - 8 Jan 2023

@obuisard change

image: node:16-alpine
to node:16-bullseye-slim and ask @HLeithner to sign the change so that the tests could run... (it took me quite some time to figure out what was wrong. What happens when someone debugs on a blank screen)

Also please add

require('dotenv').config();

after const { minifyJsCode } = require('./minify.es6.js'); in build/build-modules-js/javascript/build-com_media-js.es6.js

avatar dgrammatiko
dgrammatiko - comment - 8 Jan 2023

@obuisard I think the .drone.yml is wrong, 4.3 should have cypress not codeception for the tests. I'll say just paste the one form 4.3 directly, eg https://raw.githubusercontent.com/joomla/joomla-cms/4.3-dev/.drone.yml

avatar joomla-cms-bot joomla-cms-bot - change - 8 Jan 2023
Category Unit Tests Administration SQL com_admin Postgresql com_categories com_content com_contenthistory com_fields com_finder JavaScript com_media NPM Change com_templates com_users Language & Strings Modules Repository Front End Administration SQL com_admin Postgresql com_categories com_content com_contenthistory com_fields com_finder JavaScript com_media NPM Change com_templates com_users Language & Strings Modules Repository Front End
avatar obuisard obuisard - change - 8 Jan 2023
Labels Removed: ?
avatar obuisard
obuisard - comment - 9 Jan 2023

THANK YOU, Dimitris @dgrammatiko for helping sort this out. Thanks to Brian @brianteeman for also testing things out and all for your advice.

avatar obuisard obuisard - close - 9 Jan 2023
avatar obuisard obuisard - merge - 9 Jan 2023
avatar obuisard obuisard - change - 9 Jan 2023
Status Pending Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2023-01-09 14:38:21
Closed_By obuisard

Add a Comment

Login with GitHub to post a comment