I would propose the following handling: We consider the FTP layer a cludge to get around a broken server configuration. If you need the FTP layer, you can't use the caching, since I expect most benefits from caching to be eaten up by the FTP overhead. So we document this and add notices in global configuration and page cache plugin and would be done with this.

I don't think that the FTP layer can be implemented in any reasonable way into our caching system. For that, the filesystem caching is way to bare metal.

caching, since I expect most benefits from caching to be eaten up by the FTP overhead

Not sure why - caching is about reading existing cache files, FTP is about writing the cache files at the first render

Once the cache is generated, having FTP enabled should not slow the site at all

cludge

We either do it right or not at all - those are the only two options the project has given

I did not test - but joomla also has other caching layers like redis that don't need file system access - need to check if they work when ftp later enabled (probably do)

cludge
We either do it right or not at all - those are the only two options the project has given

Yep. And thanks to your overview of FTP issues, we either drop that rule or we drop the FTP layer. I'm torn here after wasting yesterday evening on looking at solutions to all those different issues. Looking for the others in the PLT on how to proceed.

You wasted an evening - we've wasted months

You wasted an evening - we've wasted months

And with insults like that, you will definitely get people to work even more on issues like this. With your comment people could think that you are only allowed to contribute to Joomla if you do it fulltime, but some of us have partners, family, children and work to do.

If you have a problem with me, feel free to contact me directly. You have my contact info and in worst case, my phone number is on my homepage. And as with Nikolas, you are welcome to volunteer as lead of the ATT and replace me there.

And now back to the issue at hand: @HLeithner will you decide on how to proceed here?

Maybe you can answer the question from 1 month ago from Robert

Yep. And thanks to your overview of FTP issues, we either drop that rule or we drop the FTP layer. I'm torn here after wasting yesterday evening on looking at solutions to all those different issues. Looking for the others in the PLT on how to proceed.

The problem is the basis on which the previous decision by @wilsonge was made, to continue with providing the FTP layer, was based on a flawed false assumption that it "mainly worked" and people therefore must be using it. (I cannot find his quote right now)

it wasn't a decision by george stop blaming people please.

I was not blaming anyone - I was looking for his exact quote.

Don't know what "The Problem ... decision by george" is when it's not making him responsible for it.

Anyway it was not his decision.

wow talk about toxic environment.

Listen, do you want the problem resolved or not? Simple. Make a decision.

Make a decision.

The decision is:

1. Joomla 4 drops the FTP layer

2. Joomla 4 retains the FTP layer and therefore considerable work by volunteers of a high grade need to take ownership for, and resolve the issues.

Easy decision to make.

No it's not so easy, it depends what the ftp layer should do and what's the goal it should solve

But those goals have been clearly defined since it was implemented in Joomla.

The goal is that Joomla should be able to work perfectly when on a standard set up where FTP is needed to modify files.

If Joomla cannot work perfectly, then the goal has not been achieved.

Some seem to thing that the FTP layer should be a "cludge" that semi-works, but that is not what it promotes.

The problem is that the fact that major parts of the implementation, over the years and over series of Joomla, has not worked, and people have not noticed, (because hardly anyone uses this kind of set up) so now, at the juncture of Joomla 4, its a GREAT TIME to remove the FTP layer and the baggage it brings...

The simple decision is to remove it, and lets get on with Joomla 4. Thus removing a huge barrier to the release date of Joomla 4.

Every time I work on the FTP layer, I find more issues, thus, creating a bigger release block.

I don't think that ftp layer has to work with the caching layer for example that doesn't make any sense. FTP is way to slow.

Also I don't know what was the goal of the ftp layer, from my point of view (and I know I'm one over very view people) it's a security layer if it's used correctly.

First lock your joomla installation that only ftp can modify .php file and non temporary directories like /administrator /component /template... so that you only need to use ftp in case of update or extension installation. This in combination with disabling the .php extension for "protected" directory (or better only our entry files like /index.php /admin/index.php restore.php and maybe a handful more). So that in the end joomla it self can only write to cache, tmp and log file where no php execution is allowed by the webserver. Sadly I know that we are far away from this. But reducing the ftp implementation to only the minimum what's needed to achieve this would make the ftp layer useful. Sure you also have to remove the ftp password from the config and ask for it when you need it (update, extension installation,...) but in the end you hopefully have (with some more effort in the .htaccess file/Server configuration) a two level security concept similar to windows, mac or linux with an user permission escalation systems.

Before someone writes a 10 site long manifest here, I know that this wouldn't be perfect and completely thought thru but would/could help people that likes to hardening joomla.

Looking at #33251 most of the issues should easily be solved/drop if the ftp layer is reduced of this use case but I'm pretty sure I'm alone with this opinion.

So if no one else is interested I gave up and production can decide what ever they think it's the best for Joomla.

I don't think that ftp layer has to work with the caching layer for example that doesn't make any sense. FTP is way to slow.

FTP is only used to write the cache files, and its FTPing to localhost mainly - which is fast. Once the cache files are created they are only read.

Also I don't know what was the goal of the ftp layer, from my point of view (and I know I'm one over very view people) it's a security layer if it's used correctly.

Its nothing to do with security. Its to do with overcoming the linux permissions model where an Ubuntu server will have a FTP user that uploads the files/folders and a Apache user that will run PHP. The apache user cannot write to the file system as the files/folders are owned by the FTP user.

This - in modern hosting - is no longer an issue as modern hosting run the PHP Process/Webserver with the username of the user, and FTP/sFTP is also run as that same user. Solutions such as suPHP were designed for this specific use.

This has NOTHING TO DO with securing a server.

Also I don't know what was the goal of the ftp layer,

Maybe that is the root of the problem here... that people - including those in the PLT and Joomla leadership - just dont actually understand the FTP Layer, its reason for being, and the problem it was designed to overcome. Your post - if replicated by others in the PLT - clearly shows that :-(

And before someone says that's a personal attack - its not - its a genuine observation, not personally attacking you, just saying that if you of all people dont really understand the goal of the layer, then its HIGHLY POSSIBLE there are also others, in leadership, and others, Joomla users, that dont.

ok you didn't understand me that's fine, instead of finding a solution you think I'm dumb that ok too no problem with it.

Of course I know for what the ftp layer was, what I don't know is why people still think it's used for such a use case. It's possible to transform old maybe now useless technique into a new function.

instead of finding a solution you think I'm dump

I assume you mean "dumb"... (I appreciate english is not a first language, no offence meant)

I did not say you were dumb, I quoted your own words back at you where you stated you "did not know what the goal was".

Others are calling the reason for the layer a "misconfiguration", a "botched installation" - which again clearly shows they dont understand that Its NEITHER. Its a standard way of running a server, where the FTP user and the Apache/PHP user have different roles and one cannot write to the files/folders of the other.

instead of finding a solution

I think you will find that I have invested CONSIDERABLE TIME into finding a solution, documenting the issues, and started writing PRs to resolve them.

what I don't know is why people still think it's used for such a use case.

Which is the whole point. Its been broken for so long I - and many others - doubt it is even being used at all, and therefore are advocating for it to be removed completely. and if a decision is made to keep the FTP layer then ensuring that all use-cases and interactions made by Joomla are covered.

It's possible to transform old maybe now useless technique into a new function.

FTP is dead. Its 2021. Its an insecure protocol that transfers credentials in plain text. Browsers are removing support for FTP, and serious web hosts have moved on to TLS connections, or SFTP years ago.

At the end of the day - all this talk is getting no where.

All we need is for the Joomla Project to make a clear statement. Either it wants the FTP Layer or it doesn't.

If it chooses it does, then I, and others will put our effort into resolving all the issues for Joomla 4 - and Joomla 4 will be delayed until that happens.

It really is that simple.

It really is all a waste of time to even discuss this if it doesn't even work in joomla 3 and from my tests at least it doesn't and yet no one has ever reported it.

The intention of the ftp layer when it was created was to enable Joomla to function exactly the same on a server where the web user can not write as on a server where the web user can. At the time it was created the userbase was probably 50/50 at best if not more needed it than didn't.

That was 15 years ago and a lot has changed. Other than the use case described by Harald (which seems very strange to me) I doubt very much if there are more than a handful of servers that would need to use a ftp layer.

If they do exist and there are Joomla 3 users on those servers then they currently have a terrible experience as the ftp layer is broken in many places and doesn't provide an equal service.

A decision was made 3 years ago when @roland-d proposed completely removing it from J4 but after very strong objections (at that time from @HLeithner and @nikosdion) then @wilsonge made the decision not to accept that and to continue with the ftp layer if it was working. A lot has happened in 3 years.

That was 15 years ago
A lot has happened in 3 years.

Yup, the ONLY case I can see for keeping FTP layer in 2021 is that Ubuntu installed with Apache. Apache uses www-data as a user by default, and obviously SFTP as root, or installing a FTP server with a non-root user will CREATE the problem that the FTP layer resolves.

But if you are choosing to hand build your own servers, one can (wrongly but safely) assume you should have/take responsibility yourself for fixing your permissions issues after using FTP to upload your files, then use chown to chown to www-data user.

I know of not a single professional webhost (and I deal with a LOT of web hosts!) that requires Joomla to use its FTP layer in 2021. This is due to the use of suPHP, mod_php and other new solutions that did not exist 15 years ago. These new solutions run isolated processes, running as the same username, and so no problem is created between FTP and Webserver users. These types of hosting solutions were rare 15 years ago.

@HLeithner it wasn't a decision by @wilsonge stop blaming people please.

Im not "blaming" anyone. Im directly attributing their own words to them. !!!!!!!

George, The Joomla Release Lead for Joomla 4 - stated on 3rd March 2019 that he believed it "give or take works"

George, The Joomla Release Lead for Joomla 4 - stated on 1st March 2020 - "We're keeping it."

Although a day later he stated: "admittedly I've never tested FTP Modes"

We've lost so much time over the FTP argument that this ain't no funny anymore. Joomla 4 as is right now expects a server with modern PHP (>7.4), modern MySQL (8) support for HTTP2 and SSL (although not required) and we are still discussing FTP or why dropping XHTML (although it's dead officially since 2012). HTTP3 has already landed on all browsers (some still behind a flag) but we are still talking about FTP. There must be some kind of disconnection with the current state of the web...

Best solution? The FTP support removal.
I think i have used FTP only one time (because of an amateur server) in more than 400 websites developed and 17 years of development, and i know so many devs, webmasters and professionals with thousand websites done, and no one of these have used the joomla FTP support in the past.
Let's bring Joomla in the 2021 please.

Following the projects decision to remove the FTP Layer from Joomla 4 - this can now be closed.