?
avatar PhilETaylor
PhilETaylor
2 Mar 2021

Joomla 3.9.25 "security fixes" break folder name validation because it assumes all folders must start with a-zA-Z

Equally you can no longer use a folder that starts with a period (E.g .myPrivateImages)

Equally you can no longer use a folder that starts with a non latin based char that is not a-zA-Z (E.g βetaImages)

This is a regression.

This code is at fault:

if (!preg_match("/^[A-Za-z]*$/", substr($value, 0, 1)))

To replicate

Go to Joomla Global Configuration -> Media and set the value of Path to Files Folder or Path to Images Folder to 2021images (or any string starting with an integer (or anything that is not a-zA-Z)

Click save.

Error
Invalid field: Path to Files Folder
Invalid field: Path to Images Folder

Screenshot 2021-03-02 at 18 52 40

As this code is a Rule, it will effect/affect other places where this rule is used.

avatar PhilETaylor PhilETaylor - open - 2 Mar 2021
avatar joomla-cms-bot joomla-cms-bot - change - 2 Mar 2021
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - labeled - 2 Mar 2021
avatar brianteeman
brianteeman - comment - 2 Mar 2021

@PhilETaylor its perfectly correct because it passes the tests. ;)

bad tests make bad code look good

This has directly impacted at least one of my sites

avatar PhilETaylor
PhilETaylor - comment - 2 Mar 2021

There is only one real test. The real world.

This was BLATANTLY OBVIOUSLY A BUG... I found this on code review, on a mobile phone while sat in a car. The code SHOUTS OUT how obviously wrong it is.

I don't have time to prove it, but I'm pretty sure this might also break those kind of web hosts that use an absolute path like /users/mnt/8/86/86753/html/

avatar PhilETaylor PhilETaylor - change - 2 Mar 2021
The description was changed
avatar PhilETaylor PhilETaylor - edited - 2 Mar 2021
avatar PhilETaylor PhilETaylor - change - 2 Mar 2021
The description was changed
avatar PhilETaylor PhilETaylor - edited - 2 Mar 2021
avatar infograf768
infograf768 - comment - 2 Mar 2021

Equally you can no longer use a folder that starts with a non latin based char that is not a-zA-Z (E.g βetaImages)

Imho, it’s ok to not allow utf8 characters in folders/files names.
But I agree that digits should be allowed.

avatar PhilETaylor
PhilETaylor - comment - 2 Mar 2021

I too don't like utf8 chars in folder names, but they are perfectly acceptable and valid chars for filesystems to use - people, in the real world, do use them.

Its still a regression and not “only” a security fix at the end of a series of Joomla :-(

avatar nikosdion
nikosdion - comment - 2 Mar 2021

Imho, it’s ok to not allow utf8 characters in folders/files names.
But I agree that digits should be allowed.

Those of us whose language does not use latin-1 characters beg to differ. In Greek it's εικόνες. It's not eikones; that's how we did things in the 1990s when BBS software didn't support DOS-737. It's not images either.

Likewise, I know of at least one company which uses a folder called 1option. Perfectly valid name. Not for Joomla.

If we assume that Joomla runs on a server with PHP compiled against a Unicode-capable PREG library you can use Unicode character properties to match any character that's a letter (L) or number (N) as the first character of the path. That's a poor man's workaround and breaks on older versions of PHP ostensibly still supported by Joomla. So while that would be an easy fix I wouldn't do it.

If you really want to sanitise a path without artificial restrictions you most definitely can. It goes like this.

First remove null bytes (invalid on all platforms!), convert all path separators to forward slashes (yes, they work on Windows). You need to squash doubles slashes to single slashes except if they appear in the beginning of the path.

Do the same for JPATH_SITE.

If the path begins with the converted JPATH_SITE remove the JPATH_SITE from in front of the path and trim by / on either end.

If the folder did not begin with JPATH_SITE we need to check if it's an absolute path outside the site's root: does it begin with / (*NIX absolute), // (UNC), x: (Windows) or yz: (Windows)? If so, fail with an error.

If we're still here we have a relative path. Testing where it really points to you can't use realpath because the folder might not exist yet. Trim the path by / on either end. Next up, explode() the path by separator / and do an in_array() looking for .. in there. If it exists reject it (even though it could technically still be a path inside the site, you don't want to go into full "is one path inside the other" mode — been there, done that, it's complicated and error prone, avoid avoid avoid!).

If you are on Windows (two ways to test, either the PHP OS constant or the directory separator constant) you need to sanitise the relative path. Explode by / again. Use array_map to remove from each path component all characters explicitly stated in https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file Then implode by / yet again.

You now have a clean relative path. Concatenate JPATH_SITE, / and the clean relative path to get the absolute path.

If you're on Windows check if the full path is over 260 characters; it will break on Windows versions before Windows 10 1607.

Now you have a clean absolute path.

You may want to go through that again. It's been a long day and it's nearly midnight. I may have missed something.

avatar brianteeman
brianteeman - comment - 2 Mar 2021

Did Joomla 3.9.24 work with non latin-1 folder names?

avatar PhilETaylor
PhilETaylor - comment - 2 Mar 2021

Yes, in Joomla 3.9.24 εικόνες ("images" in English) is a perfectly valid folder name that passes the file path rule, and can be used as the Path to Files Folder or Path to Images Folder

Screenshot 2021-03-02 at 22 09 21

avatar brianteeman
brianteeman - comment - 2 Mar 2021

Thanks for confirming. That makes this change a b/c break and at a minimum deserves a notification in the release announcement. I was pretty sure (but didnt have access to a server to check) as I know of at least one alternative media manager that supports utf8 files and folders

avatar HLeithner
HLeithner - comment - 3 Mar 2021

I will write a FAQ entry later today and propose a pr too.

avatar b2z
b2z - comment - 3 Mar 2021

Does it fail only for forms?

avatar ReLater
ReLater - comment - 3 Mar 2021

Does it fail only for forms?

It's a form field rule. Therefore yes. Used in 3.9.25 core (validate="filePath"): modules\mod_random_image\mod_random_image.xml and administrator\components\com_media\config.xml

avatar micker
micker - comment - 3 Mar 2021

if we have folder with only number that can broke something ?

avatar richard67
richard67 - comment - 3 Mar 2021

@micker As stated above, it only concerns validation of forms using that kind of fields. I.e. if you don't edit in such a form and save it, nothing is broken, and the site works as usual. Only when you e.g. open the form for the media manager and have a folder for files or folder for images which has a name not starting with a character, then you could not save that form without changing that folder.

Stay tuned, as Harald said, a fix will be made.

avatar micker
micker - comment - 3 Mar 2021

cool thanks

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

Stay tuned, as Harald said, a fix will be made.

Will a new Joomla release be made urgently or will this be carried over as a "known issue" - thats the most asked question elsewhere and to me.

avatar HLeithner
HLeithner - comment - 3 Mar 2021

At this time I don't see urgent reasons for a new release.

  • The filter rule was introduce in 3.9.21, I wouldn't expect a high adaption by 3rd party extensions yet.
  • It only effects changing the settings com_media and mod_random_image in core and only if your first character is not a-z character in the root folder name
  • It doesn't effect the frontend in any way
  • The workaround is really simple by removing the 4 lines
    if (!preg_match("/^[A-Za-z]*$/", substr($value, 0, 1)))
    {
    return false;
    }
    actually only the one line with the return statement.

A new PR is in progress...

In the mean time I created the FAQ entry https://docs.joomla.org/J3.x:Unable_to_use_a_none_alphabethic_first_character_in_folder_name

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

So, for those of us that need to plan, are you ruling out a Joomla 3.9.26 release today, tomorrow, this week?

People are already frightened to upgrade to Joomla 3.9.25 today and risk having to immediately have to upgrade again.

Can you be very specific please for those people watching this.

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

Conflicting messaging on Twitter - sounds like an release is imminent:

Screenshot 2021-03-03 at 16 36 00

avatar HLeithner
HLeithner - comment - 3 Mar 2021

At this time I don't see urgent reasons for a new release.

I mean no special release planned for this issue from my side at this time. Also because of the reason you stated.

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

The workaround is really simple by removing the 4 lines

So, the workaround fix, officially promoted to this issue for now, is to remove what was introduced as a "Security fix". /facepalm - again conflicting messaging.

avatar HLeithner
HLeithner - comment - 3 Mar 2021

That's a workaround not a fix. As already said we will create a pull request.

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

Well there you have it folks. No urgent release. Back to the day jobs...

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

Cross referencing another bug with this file #32577 for visibility

avatar RobertVining
RobertVining - comment - 3 Mar 2021

At this time I don't see urgent reasons for a new release.

I mean no special release planned for this issue from my side at this time. Also because of the reason you stated.

Many sites use folder names by year... images/blog/2019-01 etc.... its a common folder structure for large image sites.

Now my media manager is effectively broken with this update. That is unacceptable for a production release.

You need to fix this and release 3.9.26. TO. DAY.

That's my vote, for those that won't say it here.

Oh.... and Please with Sugar on Top. Don't want to forget my manners.

Thanks...
Your Raving Fans

avatar bembelimen
bembelimen - comment - 3 Mar 2021

Now my media manager is effectively broken with this update. That is unacceptable for a production release.

Why is your media manager broken?

avatar zero-24
zero-24 - comment - 3 Mar 2021

Many sites use folder names by year... images/blog/2019-01 etc.... its a common folder structure for large image sites.

that still works as intended. The only thing that is changed at this point is an base path for images that does not start with an a-zA-Z character while this has been discussed here this will be fixed too.

avatar RobertVining
RobertVining - comment - 3 Mar 2021

Maybe I misunderstand the issue... if I select Create New Folder in Media Manager with the folder name 2021-03 it will not save this folder name.

I'm not sure of this, as I have not tested this release on a live site with the issue. But I'll try to make time this afternoon.

avatar RobertVining
RobertVining - comment - 3 Mar 2021

Many sites use folder names by year... images/blog/2019-01 etc.... its a common folder structure for large image sites.

that still works as intended. The only thing that is changed at this point is an base path for images that does not start with an a-zA-Z character while this has been discussed here this will be fixed too.

Thanks for the clarification. I truly misunderstood. Still bad news for the use case as Phil has shown. Carry on. I'll patch my own until you fix it. Still a bad decision.

avatar richard67
richard67 - comment - 3 Mar 2021

We are no talking about "Create folder" in media manager, we are only talking about changing the "Folder for files" and "Folder for images" in the options of the media manager.

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

if I select Create New Folder in Media Manager with the folder name 2021-03 it will not save this folder name.

I had tested this, and you can successfully create folders like that still.

avatar bembelimen
bembelimen - comment - 3 Mar 2021

Again: nothing breaks for existing sites. The only issue in this case is, that if you change the base image folder in the com_media settings, (so replace "images" with something else) you can currently only use folders stating with lettert in the range of "a-z".

grafik

If you have already a folder which does not start with "a-z" (and again only the base folder counts, so "images") then you can't save the background image in mod_custom.

avatar RobertVining
RobertVining - comment - 3 Mar 2021

got it. Thanks all!

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

Again: nothing breaks for existing sites.

Factually incorrect in the real world!

If a Joomla Site ALREADY had their path set to 2020-images and then wanted to go and make a change in the Media Global Configuration to, lets say the Maximum size, then they cannot then save that new change.

Therefore you have broken this sites ability to save other changes on this page.

Screenshot 2021-03-03 at 20 44 58

avatar bembelimen
bembelimen - comment - 3 Mar 2021

Like stated above, you can't save the form with folder names starting with characters which are not a-z, that's the issue where you can use Haralds workaround.

But the media manager does not break because of the update by e.g. don't showing any images anymore. So after the update the website continue working, just saving the option does not work with... yeah I repeat myself...

avatar PhilETaylor
PhilETaylor - comment - 3 Mar 2021

Correct. We agree that it breaks existing sites functionality.

avatar hvdmeer
hvdmeer - comment - 4 Mar 2021

Conflicting messaging on Twitter - sounds like an release is imminent:

Screenshot 2021-03-03 at 16 36 00

I posted that and I can assure you that if we post about a fix we don't mean a release.
When we use the word release in our post, then we do.

Sorry that this caused confusion.

avatar WebSpain
WebSpain - comment - 4 Mar 2021

Conflicting messaging on Twitter - sounds like an release is imminent:
Screenshot 2021-03-03 at 16 36 00

I posted that and I can assure you that if we post about a fix we don't mean a release.
When we use the word release in our post, then we do.

Sorry that this caused confusion.

So when you say a fix do you mean you'll be patching the existing 3.9.25 version that's currently available?

avatar PhilETaylor
PhilETaylor - comment - 4 Mar 2021

I posted that and I can assure you that if we post about a fix we don't mean a release.

"Fixes" are not done by a "Release Team". Releases are done by a release team

"Stay tuned" normally means something is actually happening and will be announced very very soon in English.

You will have to excuse people for assuming, incorrectly, that this backward compatibility break was being fixed quickly by a release team with a view to being released by said team in a timely manner, especially as you announced the release team were working on a fix - as Joomla has history of having to make multiple releases after screwing up security merges at the last moment before a release. Most people now wait days before upgrading because of this and it has now come to harm Joomla where people don't trust releases to be the "last release that day"...

When we use the word release in our post, then we do.

The word "release" was in your post hahahahahah

So when you say a fix do you mean you'll be patching the existing 3.9.25 version that's currently available?

What he actually means is a "Workaround" is currently available, until Joomla 3.9.26 is eventually released with a fix,

The workaround, which is currently promoted in the url below, is removing the lines of code that were added to "address a security issue" so that existing sites can operate as they did before, without backward compatibility breaking changes.

see: https://docs.joomla.org/J3.x:Unable_to_use_a_none_alphabethic_first_character_in_folder_name

The "Fix" that was announced int he tweet has still not been made public.

avatar WebSpain
WebSpain - comment - 4 Mar 2021

Thanks for the clarification Phil.

avatar RobertVining
RobertVining - comment - 4 Mar 2021

While you guys debate what's right or wrong... we have a Joomla! Development Strategy that outlines what we do in this situation.

Let me quote it for you.

"6.3 Regressions
There will no doubt be occasions when a release unintentionally breaks backward compatibility. If one or more such backward compatibility regressions are found within a major series they will be fixed by making a patch release as soon as possible after their discovery."

Found here https://developer.joomla.org/development-strategy.html

avatar PhilETaylor
PhilETaylor - comment - 4 Mar 2021

LMAO

I don't have time to deconstruct all the ways that policy is not followed. I have a day job.

" full backward compatibility can be expected within a major series. Backward compatibility may only be broken when a new major series is started."

Cough....

"All PHP code in the /libraries folder which is not flagged as private is considered to be part of the Joomla API and subject to backwards compatibility constraints."

Joomla 3.9.24 was announced as the last in the Joomla 3 series, unless a major security security issue was found. However Joomla 3.9.25 only fixed Low Severity (Difficult to perform. Relies on sensitive information or requires special circumstances to perform.) security issues... and introduced backward compatibility issues. The Release was sloppy and did not follow the release checklists, also releasing __DEPLOY_VERSION__ placeholders into the final build... I get that releasing software is hard, but this is why there are policies and checklists and procedures and automation to help.

This change (the a-zA-Z change) was blatantly obvious to anyone that it would cause issues - but still the JSST did not see it... This is the team charged with keeping us all secure...

avatar mrownicki
mrownicki - comment - 5 Mar 2021

It's 2021 year, can't do a function that changes utf-8 to ascii on the fly. for create new path. What is user interaction for?
People don't want to see message people want using cms :) Not bothering with paths.

By the way why not fix files globally adding and creating folders. Natively new installations run on the ascii character set, where the user can switch to UTF8

avatar gwsdesk
gwsdesk - comment - 8 Mar 2021

Why are we so stupid and ignorant @HLeithner not to simply do a quick release where thousands of website are broken because of this? i personally can add or remove line in a piece of code but many users cannot or don't care. A reference to a FAQ does not work since we all know that nobody reads the notes or FAQ...." If I have a problem how should I know that I have to look at FAQ's?" These are not posted on the Joomla forums so most users have no way to find out why the release broke their site

Get a release with a fix out ...you are in charge so do something for this community please? Typing many reactions here and elsewhere takes you more time then simply changing a few lines of code and releasing the patch! Get over your pride and spend 30 minutes to get this issue solved!!!

avatar sozzled
sozzled - comment - 8 Mar 2021

Thanks for this discussion. Like many other people, I have a lot of websites (well, actually, that's not true: I've reduced the number of websites that I maintain by more than 75% over the last six months because it just wasn't worth my time maintaining them given the current pathetic inactivity within the J! project that would otherwise inspire me to persevere) that use folder names containing numeric characters.

It's useful, when I'm writing articles, to upload images into folders that have names like ../0321, for example (where 03 is March and 21 is 2021 if I'm writing an article to be published in March 2021) or any other folder of the form ../mmyy.

However, I have never changed the Path to Files or Path to Image Files settings as shown in the screenshot above. So, while I've happily updated my [few, remaining J!] websites to J! 3.9.25, I haven't personally experienced this problem but I do understand that it could affect a sizeable number of J! website owners who want to practise their webcraft in their own particular way.

For a number of reasons, I have stayed clear of public commentary on The Joomla! Forum™—particularly after the put-down I received when I commented about J! 3.9.24. So, to cut to the chase: The Joomla! Forum™ is probably the worst place in the world to post "FAQs", other helpful information or feedback about new releases of J! because no-one reads it and the development team (what's left of it) don't visit it.

avatar brianteeman
brianteeman - comment - 8 Mar 2021

The title of this post is leading to some confusion. The problem is that the foldername for "the path to images folder" can not begin with anything other than a-zA-Z. There is NO problem at all with folder names such as images/2021-01

Also as stated in the FAQ just by upgrading to 3.9.25 would not have broken any site at all. The problem would only occur when you tried to change the options for the media component.

Obviously the change to the rules for naming the base folder for images is an error and is a break in backwards compatibility but I doubt very much if even a host such as Leo has thousands of sites that are impacted.

Are you really trying to say that you have thousands of sites that have done all of the following

  1. Changed the default path to images folder
  2. Changed it to a folder beginning with a number or other non a-zA-Z character
  3. Upgraded to 3.9.25
  4. Tried to change the options for the media manager

Because only on a site that has done all of those will this unfortunate bug have any impact

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

@sozzled The official Post Release FAQ is (should) always be linked on the announcement of the release, eg on the page https://www.joomla.org/announcements/release-news/5834-joomla-3-9-25.html.

Its the small text with a link to a wiki page (that's normally forgotten to be created, if there were no known issues at the time of release

Questions? See the documentation wiki for FAQ’s regarding the 3.9.25 release.

The link on this occasion went/goes to https://docs.joomla.org/Category:Version_3.9.25_FAQ

Any commentary in the forum is just that, commentary :)

Edit: I should point out that the FAQ about THIS ISSUE was NOT released at the same time as Joomla 3.9.25, it was an unknown issue, and that the FAQ was added some time (day after?) the release once this issue was reported.


The point is this is a backwards compatibility break right at the end of a series, after an announcement was made that no more releases were going to be made unless SERIOUS security issues were identified. The 9 security fixes were all "Low Severity" but now we have at least 2 regressions breaking backwards compatibility (and 5 typos in a tiny release).

The code was not just wrong, it was BLATANTLY wrong, and any peer review of the code, before release, should have picked this up. Its not a complex buried deep change, it was glaringly wrong from the outset.


Its not just about the folder name. Because the change stops the whole page of options being saved. So any change to the options on the same page (such as the maximum size) can no longer be saved.


There was also further initial confusion when someone tweeted from the official account saying the "Release team were working on a fix"... which a native English speaker (several of us) assumed that a quick fix, and second release on the same day was going to happen (as Joomla has a long history of this kind of thing, double releases) - and now people are wary about applying security updates immediately, because they are scared that they will have to apply yet another release a few hours later.


Now we just have to be patient and wait for the JSST and the Joomla Project to rewrite the code, to fulfil fixing whatever "security issue" the JSST were trying to "fix" with this patch, that I, and many of us are not party to the full disclosure of information.

I would say Joomla could learn from his mistake, but history keeps repeating itself. What is done is done, and thankfully it doesn't effect live sites as bad as the first reports, including mine, were led to believe. This is more by luck than by design though.

avatar softforge
softforge - comment - 8 Mar 2021

I have 120 + sites and none were affected.
They were however badly affected by third party software breaks which had nothing to do with Joomla core changes.
People produce comercial extensions and issues happen even though they are paid to build these extensions but I didn't see much in the way of complaints.
Core is all volunteers who are giving of their time for free yet are jumped on and chastised should something unplanned happen. I do not understand why people who make their living off Joomla are so quick to hound and insult those who have spent so much time producing that platform.
There is an old saying "Cutting your nose off to spite your face"

Wiki - The expression has since become a blanket term for (often unwise) self-destructive actions motivated purely by anger or desire for revenge. For example, if a man was angered by his wife, he might burn down their house to punish her; however, burning down her house would also mean burning down his, along with all of their possessions.

Would it not be wiser to privately lend a hand to fix the issues, and help your own projects be more successful rather than publicly praise your hindsight but damage the project your own incomes are based on?

There is no virtue in this bitterness, just hollow victories.

avatar sozzled
sozzled - comment - 8 Mar 2021

Thanks for the clarification, Phil [Taylor]. I agree with you that it might be an idea (when particular issues that may concern some J! website owners are identified) for someone to insert, or update, the forum announcement with new, relevant, "FAQ"-related information. That's just one of those things that happens (or doesn't happen), I guess.

I agree with you that any public commentary on a public forum like The Joomla! Forum™, for instance, is just that ... commentary. However, when public commentary about lack of appreciation shown by "the J! development team" (what little remains of it) is removed from public forums, like The Joomla! Forum™, it makes one wonder why we bother writing anything at all. Because, when we do comment, our personal views are removed for being "irrelevant" or "off-topic"!

As for Phil [Walton] who writes:

Would it not be wiser to privately lend a hand to fix the issues ...

Why do things have to be done "privately"? What's the big secret? Sorry, but I just don't understand this commentary.

avatar softforge
softforge - comment - 8 Mar 2021

No secrecy is needed, if you want to help and blow a trumpet at the same time then do. Doing things without a song and dance does not mean secrecy, just modesty

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

Keep drinking the Kool-aid @softforge ...

I do not understand why people who make their living off Joomla are so quick to hound and insult those who have spent so much time producing that platform.

LMAO... I do not make my living off Joomla at all. I make it off my expertise, ability and experience. If I relied on Joomla for my income, then my income would by dying like the project is. As my public accounts show - in public - I do more than just Joomla, and my income in not in mass decline.

I, like others, have contributed years to this project, right back to the beginning of Mambo Site Server, contributing code, features, and serving in many roles, even in the JSST.

I also, by Joomla's own measurement of "user activity" contribute in the top 6...

Screenshot 2021-03-08 at 19 27 31

Screenshot 2021-03-08 at 19 28 37

Attacking people because their day job is "semi" related to a project is pathetic. I contribute to the Symfony project and use Symfony code day in and day out and have NEVER ONCE been attacked for contributing, commenting, raising bugs, or pointing things out.

This was broken by the Joomla Security Strike Team in response to a reported security vulnerability the full details of which have not been provided in public or in private. Therefore no one can "fix" this issue without potentially introducing the same security vulnerability that these lines of code were added to prevent. We wait on that team to implement a fix, as Harold has already stated he would do - 6 days ago.

If the JSST would like to publicly or privately share the vulnerability then I, and many others would be happy to help.

There is also a line between " something unplanned " and "something blatantly wrong that should have been caught by simple peer review, of a team of security wise, experienced, developers, in a security team of a project that powers <3% of the internet.

"Cutting your nose off to spite your face"

If reporting bugs in releases is considered a bad thing, if you are perceived to make some kind of living from anything Joomla related, then Joomla has a bigger issue...

I do not understand why people who make their living off Joomla are so quick to hound and insult

I don't see anyone "hounding and insulting" ??

Core is all volunteers who are giving of their time for free

There has not been a thing as "core developers" since way back in Mambo days when there were 5-6 of us...

We are all developers of Joomla now, just login, and contribute... simple. What you are trying to allude to is that a tiny amount of people do a high proportion of the perceived "work".

The days of rolling out the "Core is all volunteers who are giving of their time for free" mantra are over. Either this project takes security seriously or it doesn't. Its called responsibility. You don't get to have a position within a team, and not have a responsibility to the project. Just because you "volunteer" doesn't mean you are absolved of responsibility. Ive heard this same argument "but we are just volunteers" too many times...

The JSST have the whole responsibility for the security of the Joomla Project. Mistakes this blatant should not be made.

There is no bitterness here at all. These are just all facts. They are only construed as bitterness because you see them next to my name. If I used one of my many aliases then I would not get half the hate I do here.

Would it not be wiser to privately lend a hand to fix the issues ...

I literally spent years of my life doing that for this project in the JSST. Stuff you could only dream of.

No secrecy is needed, if you want to help and blow a trumpet at the same time then do. Doing things without a song and dance does not mean secrecy, just modesty

Song and dance? I reported a bug in the tracker... that's all I did... pfffff.

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

The only time the words "idiot" and "incompetent" were used in this issue... is in your [now deleted] post @drmenzelit ...

avatar drmenzelit
drmenzelit - comment - 8 Mar 2021

You are right... I apologize

avatar drmenzelit
drmenzelit - comment - 8 Mar 2021

I have the feeling that everything one want to say is misinterpreted. I have not personally offended anyone with my post. Now there is a comment without context that puts my deleted post in a false light. I give up.

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

Maybe you should not have deleted your post then:

@drmenzelit The problem is the bad habit of shouting everything out loud on Twitter instead of talking to each other normally. There's no need to call others idiots and incompetents right away. It may be that someone have already pointed out a problem that was not immediately fixed or taken seriously, but it is still not right to immediately insult others. Everyone makes mistakes and it should be possible to communicate in a positive way, even if it's for the hundredth time.
But maybe it's just my way to always look for the positive in people and not immediately think something was done wrong on purpose.

avatar sozzled
sozzled - comment - 8 Mar 2021

@PhilETaylor wrote:

The days of rolling out the "Core is all volunteers who are giving of their time for free" mantra are over. Either this project takes security seriously or it doesn't. Its called responsibility. The JSST have the whole responsibility for the security of the Joomla Project.

Leaving aside some exaggerated/alarmist propositions about how/why/whether the JSST have responsibility over what changes are introduced into new releases of J!—and I have little interest in participating in that commentary—I have twice made unchallenged assertions about the current involvement by the Joomla development team, viz. the development team seems to have evaporated or become impotent (whether it's because of a lack of volunteers or whether it's because there's a lack of enthusiasm).

Phil [Taylor] slightly exaggerates the "less than 3% of the internet" powered by J! websites ... but only slightly: https://w3techs.com/technologies/history_overview/content_management/ms/y

But that's not my point. I would like to know just how many people are still currently and actively involved in maintaining J! 3.x. That shouldn't be a difficult question to answer. A couple of years ago I would have estimated that there may have been as many as twenty or more people who were directly contributing to PRs that made their way into the CMS. We could add another half-a-dozen people who actually tested these things, a couple more who wrote documentation, one (or two) who released these things and the hundreds of people who found fault with the new versions after they were released. Of course, the last-mentioned of these never participate on GitHub and, conversely, the developers never bother to read about those matters unless they're posted on GitHub. This seems to me to be the main problem: the developers and the users are simply disconnected from one another and the developers have disengaged from that conversation.

So, I'd like to know, how many people are still currently and actively involved in maintaining J! 3.x?

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

This is probably better topic for the Discussions or the Forum, rather than appending this Issue.

Leaving aside some exaggerated/alarmist propositions about how/why/whether the JSST have responsibility over what changes are introduced into new releases of J!

Not exaggerated at all.

They oversee, are tasked with, improving and managing security for Joomla. As the following quotes from https://developer.joomla.org/security.html state:

  • "the Joomla! Security Strike Team (JSST) oversees the project's security issues"

  • "The JSST is called a strike team because it is a collection of developers and security experts tasked with improving and managing security for Joomla.

  • _The Goals of the JSST are:

-- Investigate and respond to reported vulnerabilities in the Joomla! CMS, Framework, and joomla.org websites.
-- Execute code reviews prior to release to identify new vulnerabilities.
-- Provide public presence regarding security issues.
-- Help the community understand Joomla! security._


Phil [Taylor] slightly exaggerates the "less than 3% of the internet" powered by J! websites

The 3.4% on the link you provide is the Market Share for Content Management systems.

Joomla is reported to have 2.1% of all the websites that are monitored by W3Techs (Which is often wrongly stated as "2.1% of all sites on the internet") ... which is 3.4% market share in the Content Management systems category.

My choice to use "<3% of the internet" relates to the fact that 2.1% is less than the "more than 3 percent of the Web running on Joomla!" that is the marketing tag line used by the official Joomla Project and is a clear sign of the decline of use of Joomla globally. The historical trend is factually a decline https://w3techs.com/technologies/details/cm-joomla with 24.2% of Joomla sites not even bothering to upgrade to Joomla 3 series.

I would like to know just how many people are still currently and actively involved in maintaining J! 3.x.

Well it depends on your definition of "actively involved", as Joomla has a floating way of defining "contributions" to fit the agenda at the time of writing. For example, there are those that spend a lot of time in the Issues, and then there are people that code. A few people do both. The only public stats the Joomla Project publishes is the User Activity Trackers and ACtifit snapshots in the Project menu dropdown here https://issues.joomla.org/tracker/joomla-cms

A couple of years ago I would have estimated that there may have been as many as twenty or more people who were directly contributing to PRs that made their way into the CMS. We could add another half-a-dozen people who actually tested these things, a couple more who wrote documentation, one (or two) who released these things and the hundreds of people who found fault with the new versions after they were released.

My gut feeling which counts for nothing, but you asked... - totally unscientific and just what it "feels like"

Within GitHub:
1-10 people actively writing code for Joomla 3.9 series (who are not doing so as they role in JSST states)
1-10, people writing code for Joomla 3.10
10-20 people writing code for Joomla 4, which swells and shrinks in waves, the period of a wave also varies greatly. There seems to be a big push, and then silence for months. Some contributions (like mine) are quality related (and a million spell checks haha) and some others are bigger, and yet some others (like from Brian) are to remove features from Joomla 4 because they are not ready.
1 Person merging Joomla 3 changes to Joomla 4 (George)
4-6 people merging code PRs into Joomla 4
5-20 people testing PR's for Joomla 4 and marking their test results. (16 people in the last 7 days have received 5 or more "test points" in the tracker)
10-20 people writing meaningful Issue Reports for Joomla 4
As for documentation, I don't do wiki but Im sure there is a way to find a number of active contributors.
But as I say, these are only SOME ways of contributing to the project...

So, I'd like to know, how many people are still currently and actively involved in maintaining J! 3.x?

I doubt you need to take your socks off to count on your fingers and toes the number of people, and I would not be surprised because Joomla 3.9 has already been announced as being at the end of the line and Joomla 3.9.24 was spoken of as being the last 3.9.x version, unless serious security issues were found, and there was no expectation that Joomla 3.9.25 would ever be released (hence why people were surprised when it was, and then that it only contained low security issues, backward compatibility breaks, __DEPLOY_VERSION__ placeholders, and a myriad of typos)

The root issue that is killing Joomla, and this has been endlessly talked about by other commenters on the web, is the fact that Joomla 4 has been in development for so long without a stable release, and Joomla 3 has been "stable" for so long. A lot of people have given up waiting for Joomla 4 and moved on to other platforms. But again that is only an opinion from one idiot, me.

avatar sozzled
sozzled - comment - 8 Mar 2021

Thanks, @PhilETaylor. That was neat and uncomplicated. I agree that we've moved away from discussing one issue (among countless others) that relates to the design, development, testing and implementation of a change (or changes) to a new version of J!. And, given that J! 4 is itself a basket-case (and many of the security and remedial changes that have taken place with J! 3.x will probably be forgotten or become a patchwork-quilt of fixer-uppers), I don't think it's just that the five-year epic journey with J! 4. is the root cause that's turned people away from J!.

I believe that the root cause is the disengagement between the code, the culture, the community and OSM that supposedly supports these areas. I think we've become distanced from the idealism that led to the foundation of Joomla! I believe that the real cause for the problems that people now encounter with Joomla lie only in part with an inability to redress the management of the project. I believe that a culture—if that's the right term—has developed and it's led to schisms within the community.

The easiest-to-identify schism is the disconnect between the developers and the "end users". Sure, the developers rarely receive thanks for what they've done but, by the same token, it's us "end users" who are similarly unappreciated for getting J! to where it is ... even if only in terms of "market share". What I would like to see is a way to mend the disconnect between the developers and the rest of us. I believe it is totally unfair on the rest of us to be coerced into using GitHub simply because the developers eschew The Joomla! Forum™. I believe the root cause of most of our problems is the fact that the code, the community and the culture (and OSM that exists behind these things) are travelling along separate pathways.

If the end users—the rest of us—see more participation from the developers of the code (as well as from OSM) in places that "the rest of us" normally visit, then perhaps we'd see an end to some of the name-calling. It's one thing to label something as disagreeable as this specific discussion has been about in a back-woods setting such as GitHub where possibly a handful of people may spend a few moments reading some of the nonsense we've written, but it's another thing entirely if these kinds of general discussions about the malaise that's infected the project were to be discussed in a more public forum. The question for me (and probably for a lot of others) is where is the most fitting place to have that open Q&A?

avatar brianteeman
brianteeman - comment - 8 Mar 2021

many of the security and remedial changes that have taken place with J! 3.x will probably be forgotten or become a patchwork-quilt of fixer-uppers)

Sad that you feel that way. @wilsonge specifically and many others work very hard to keep the changes in J3 merged upstream. Its a technically difficult task and very time consuming as each change has to be reviewed and often modified to the J4 approach. This is usually done in batches and you will rarely see any evidence other than the occasional PR containing all the changes at that time. As far as I am aware everything relevant has been merged up into J4 from everything up to 3.9.23.

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

I believe that the root cause is the disengagement between the code, the culture, the community and OSM that supposedly supports these areas

Yes probably, that is a lack of leadership.

Look, I try my best to stick to the code. The code is what Im comfortable with and TRY to stay away from the politics of the running of the project and the leadership vacations to the beach. Im a autistic geek who is just an idiot and not well liked. I get that. I know my place. OSM was, is, and always has been a joke. And now I hear the project has huge financial issues whereas it used to make millions and have millions in the bank.... but I digress.

Users who stray into GitHub are entering a lottery, I know people have emailed me to say they are physically scared to log an issue - including female previous post holders in Joomla Leadership. This should not be the case but it sadly is.

I know developers that would never go into the forum, as that is where users ask (they perceive) "stupid questions".

I have heard of some perceived "core" developers of Joomla, who don't even use Joomla themselves for projects.

There IS a disconnect between these circles of people. I agree.

@wilsonge specifically and many others work very hard to keep the changes in J3 merged upstream.

Far too much for one person, and a burden that could be easily shared. @wilsonge has done incredible, amazing to get it this far.

avatar sozzled
sozzled - comment - 8 Mar 2021

Thanks, @brianteeman. I was probably out of line to have posted what I did because I was reflecting on another put-down I received—I forget from whom (but it wasn't you)—when I asked how remedial changes introduced into J! 3.9 over the past couple of years would find their way into J! 3.10 and J! 4. At the time I asked that question—six months ago, at least—I was given short shrift and basically told, "we'll do our best, we're not promising anything, but they're the last items on our 'to do' list". So, when I moaned about it here, I wasn't doing this discussion any great service.

There are a few things that Brian and I disagree with (and we agree that we disagree of things from time to time) but I have an abiding respect for Brian and we actually agree on more things than people may give us credit.

The problem in making sense of what @PhilETaylor has written, trying to "stick to the code ... and [staying clear of] the politics" is that once one mentions OSM as being "a joke" and the project has [however one measures these things] "financial issues" we've entered a political arena. The problem isn't a political one; it's not even problem about QA or peer review.

@PhilETaylor wrote:

I know developers that would never go into the forum, as that is where users ask (they perceive) "stupid questions".

Isn't that what I've been saying? The developers have disengaged themselves from the rest of us. By the same token, "the rest of us" don't venture in GitHub because they get unfathomable technical "explanations" as to our ignorance about how things work. So, to give a non-technical explanation to the developers about why things don't work, please look around you, look outside GitHub, and engage with the rest of us.

avatar richard67
richard67 - comment - 8 Mar 2021

@PhilETaylor You wrote above: The only time the words "idiot" and "incompetent" were used in this issue .... That might be right for this issue and for you.

Unfortunately I don't have the screenshots here now which I've seen from Twitter (I am not on Twitter myself so can't search that stuff now) in which we all who contribute here, not only the maintainers, were called "amateurs", "idiots", "morons" and so on by certain people (not you), but you retweeted, maybe not using the same embarrassing words, but at least supporting them in their meaning.

After reading all these accuses against "the developers" I ask myself I what I've done here in the past 6 years all was wrong. Sorry for that if so. Just let me know if people like me are bad for the project, and I will stop with all I do. I did all I did because I loved Joomla and wanted to contribute to it to make it better, not because I needed some kind of hobby, so if it turns out that all that was shit and bad for Joomla, I will not insist to continue with it.

Thanks for ruining my motivation for now.

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

by certain people (not you), but you retweeted, maybe not using the same embarrassing words,

Is a retweet an endorsement? You have much to learn about twitter. Once you do you actual research I'll accept your apology.

My timeline is public for 14 days here https://twitter.com/myPhilTaylor/with_replies

avatar richard67
richard67 - comment - 8 Mar 2021

Once you do you actual research I'll accept your apology.

@PhilETaylor There is nothing I have to apologize for.

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

So you are free to accuse me of things I have not done... got it...

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

so I did your searching for you....

My only three retweets about Joomla (most of my recent tweeting has been about local island issues!)

Screenshot 2021-03-08 at 23 25 33

Yup I totally agree....

Screenshot 2021-03-08 at 23 26 04

Retweeting the official account which said the release team were working on a patch 6 days ago...

Screenshot 2021-03-08 at 23 26 56

Important information for Akeeba, the number 1 extension for Joomla, retweeted because it was important.

So @richard67 what EXACTLY are you accusing me of?

avatar richard67
richard67 - comment - 8 Mar 2021

@PhilETaylor Could be I was wrong with you contributing to that developers bashing personally. Sorry for that. Could be that after all I had to read at diverse places made me end here in an emotional state which makes me bash back the wrong people. I don't know. Maybe I should have not said anything.

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

All I have done is factually report the bugs in Joomla 3.9.25, from the small amount of changes made in the release, there were considerable mistakes made and backward incompatibilities introduced.

I have also contributed no less 5 pull requests against the changes made in Joomla 3.9.25 and reported 2 backward incompatibilities, and one sloppy mess that has lead to factually incorrect @since statements.

And all Im getting is flak... on my birthday...

Issues:
#32578
#32577
#32567

And pull requests:
#32580
#32575
#32570
#32581
#32579

I believe this bug (that this comment is on) was preventable, with peer review, and should never have been able to have been released, but as Brian is stating, **** happens. It was blatantly wrong from the start and I discovered it reading the release changes while on an iPhone sat in a school car park collecting my daughter from school.

I really struggle to understand how the JSST thought a-zA-Z was acceptable, but then Im not privy to the actual security vulnerability reported. If this had been Team reviewed, one would have hoped that it would have been flagged.

The "workaround" promoted by JSST team members to "just delete" the lines added - when they were added as part of a security release I do believe is a bad statement for Joomla to be making. After all, if it was no less secure to remove the lines why were they added in the first place.

The evidence is all there that this was a sloppy release. I should not get flak for highlighting that, and reporting the issues and communicating them widely.

Yes I converse on Twitter with people that have been banished from this project, who have been treated badly by this project, who have been banned by this project, whose mental health has suffered, and people who have been badly let down by the leadership in this project. Sometimes they have valid points to make, with their own colourful way of putting things. Some in this project have a huge debt of thanks to pay to some of these people. Joomla would not be where it is today without their contributions.

avatar richard67
richard67 - comment - 8 Mar 2021

Yes, sorry again for my off topic post.

avatar PhilETaylor
PhilETaylor - comment - 8 Mar 2021

lol most of this issue is off topic now... never mind...

avatar rdeutz
rdeutz - comment - 9 Mar 2021

I locked this now, anything what needs to be said is said.

@gwsdesk calling other people "stupid and ignorant" is not an acceptable beauvoir.

avatar richard67 richard67 - change - 17 Mar 2021
Status New Closed
Closed_Date 0000-00-00 00:00:00 2021-03-17 15:47:33
Closed_By richard67
avatar richard67 richard67 - close - 17 Mar 2021
avatar richard67
richard67 - comment - 17 Mar 2021

Closing as having a pull request. Please test #32718 . Thanks in advance.

Add a Comment

Login with GitHub to post a comment