I believe one win win sollution is automatic updates for minor releases with an opt in option.

1. The majority of simple sites don't core hack and most of them are left "unmaintened" so the security gain would be enormous and they should ofc opt-in this feature.
2. The ones that core hack they have the option to not opt in this feature. Since they core hack they should probably be able to regularly manually update their sites.
3. There will be a bigger burden to more thoroughly test the minor releases for bc changes but this is actually not too bad. It can be an advantage since all the extensions developers (that support the joomla auto-update feature) will check in time.
4. When the bad thing will happen (cause sometime it will ) we have the option to release a next minor version soon.
5. Backups can be triggered from software like akkeba before update for more needy sites that cannot afford 2-3 days for waitin next release.
6. And ofc for sites with mainteners - administrators that want to test before applying it's opt-in so they can just don't use this feature.
   (i cannot talk about package validation. I understand the risk but i guess the wordpress hasn't taken this measure yet(50% of the market) so i guess it's not a blocker for this?)

Last but maybe the most important thing about this is that this feature is a

killer feature for joomla 4

I believe it's more marketing usable than bs4, namespacing, or our new template...
We talk about security.
We will be able to make sites for costumers that cannot afford much and these sites are gonna be relatively more secure for a much bigger time than now.

I forgot to say we need a very good message (like the post install?) to explain what is this feature the "dangers" of using it, warn that some extensions or site builders that core hack should not use it, the advice of it's always good to keep backups etc...

In the context of Joomla (and every SemVer following project), a "minor" release is any .0 feature release (i.e. 3.8.0), whereas WordPress' are every non-.0 release. More often than not these are the releases people groan about most and should not be the releases we be auto updating to at first. So presumably you're talking about patch releases here.

I understand the risk but i guess the wordpress hasn't taken this measure yet(50% of the market) so i guess it's not a blocker for this?

In today's market it does need to be a blocker. Imagine for whatever reason enough of the joomla.org architecture gets hacked and someone is able to fully release a package to all channels (by my count that's at least 4 different systems if you want to do it in a way where it doesn't come across as suspicious if you aren't following this GitHub repo). Without signed releases, there is no way for sites that are out there today to reject receiving the update (and with them not only would you have to compromise our hosting architecture but at least one release lead's signing key). A checksum validation at least makes MITM attacks more complex to achieve but doesn't alleviate the risk. You mention at the end security needs to be a focus point, this is the biggest security concern with allowing automatic updates; allowing unvalidated code to be consumed and executed.

Yes i was referring to patch releases for auto update. 4.0 to 4.1 will have to be manual.
As for the unsigned releases you are by far more expert to see the possibility of a compromise. I just made a simple naive thought that if wordpress keeps doing it while this danger exists maybe we could tolerate it too. (I don't believe wordpress have much more super secure servers than Joomla's).
Ofc it's ideal to address it and be more secured than wp autoupdatesystem. However if due to the difficulty of addressing this specific issue, we stop making this feature then i think we should't let it be a blocker
(sorry for my english i hope you understand what i am trying to say :)

It's not a difficult item to address at all. The checksum validation code adds ~125 lines to our update systems for both core and extensions. The patch for WordPress adding signature validations adds ~50 lines. Their problem is they're more focused on UI features than architectural issues like this one because they believe users should never be inconvenienced by technical issues (going so far as to disable HTTPS and SSL validation and falling back to plain HTTP if a server can't securely connect to their systems to download updates).

so just remove this point from rfc :) It's obvious that it should be implemented if it's easily done and more secure:)

The main point in putting it is that as of this moment we do not have code doing any form of package validation in a release branch. Even when the checksum code does merge for 3.9 or 4.0, that's still only part of the equation and the signing part needs to be implemented then (as well as internal workflow changes related to those features).

IMO Automatic updates make sense if we can provide at the same time a one-click "disaster recovery" feature, regardless of the "security only releases" or "signed releases"

p.s.
are we still in time to add GSOC 18 project for implement the signed releases ?

are we still in time to add GSOC 18 project for implement the signed releases ?

That particular task on its own couldn't fill an entire GSoC project. Like I pointed out, it's a 50 line diff in WordPress' code, if our diff is much bigger than that we've got problems ?

IMO Automatic updates make sense if we can provide at the same time a one-click "disaster recovery" feature

We can't. That means core has to have a robust backup and restore engine in core. IMO that is beyond scope of the core distribution (and yes I know there are people pushing for it and that there are some tools already in the core API which could support import/export). The system has to run in a way that allows existing extensions to hook the auto-update process (similar to how Akeeba Backup hooks the core updates now), but beyond that I'm not convinced it needs to be on our roadmap to have backup/restore operations to the extent this calls for as a core feature.

Anyone who knows me knows I'm no fan of automatic updates,

I share that view for automatic updates. As you pointed out there would be a big task to support every release until the series hits eol with sec updates.

To me this would mean:
3.8.0 is released & installed by the user.
3.8.1 no sec updates
3.8.2 no sec update
3.8.3 no sec update
3.8.4 no sec update
3.8.5 sec updaten
=> 3.8.0.1, 3.8.1.1, 3.8.2.1, 3.8.3.1, 3.8.4.1 needs to be tested packed, tested and released.

And our major versions has a long support time. So there is also a 3.0.0 (released 27 September 2012) version out there. So we also need to release a 3.0.0.xx and every version between that. That all include testing and we still need to build and support any versions. I don't see who has the time to do all the needed work like test all versions starting with 3.0.0 against the patched sec issue and write patches for software build in 2012 in the needed quality and time (maybe just 1-3 Days if you have luck and depending on the issue).

Also if we patch something automatic we get into a "joomla broke my site" problem because of the problems you have also pointed out above. We also remember extensions or tempaltes that need a update first before we can apply a core joomla update without problem. And in the most cases this is not out before the core update. In some cases this needs weeks until the update is out. Do you want to automatic install that 3rd party update too?

To me the first step would be signed releases alone without any automatic behaviors. So we start securing our update channel and building the infrastructure around it.
If that is working we can think of the next steps in that direction if we want to take them.

And our major versions has a long support time. So there is also a 3.0.0 (released 27 September 2012) version out there. So we also need to release a 3.0.0.xx and every version between that. That all include testing and we still need to build and support any versions. I don't see who has the time to do all the needed work like test all versions starting with 3.0.0 against the patched sec issue and write patches for software build in 2012 in the needed quality and time (maybe just 1-3 Days if you have luck and depending on the issue).

There is an easy answer to this and i don't think anyone (website builder) has the right to get furious about this.

3.9 release blog---> We are glad to introduce this long awaited great feature bla bla bla ....
... for technical reasons in order for joomla sites to be benefited from this, the update to 3.9 must be done manually. From 3.9 and going forward the ones that opt in for this feature will be able to auto update.

I was thinking why this issue mbabker raised hasn't taken much discussion. Am i the only one that thinks it's long awaited? Then i figured an answer. Here in github the ones that watch it are developers or web site builders that make "big" sites. By the term big i mean a site for someone that will be able to cover a not so small fee per annual or monthly basis for an administrator to maintain his site. Well, i believe (i don't have facts to prove) that many many (maybe the majority?) of joomla sites don't have this capability (of some administrator to be there when a security patch is released to do it in under 3 hours...).
The viable sollution right now is to go for wp for these types for sites. Why???
Wp has set a standard there. We can do better!
Let's not reply to this with our nose up and say "well these types of sites deserve only wp, not the superior joomla."
It's not true! Joomla can perform equally or better for small sites too. It just has too high maintenance cost. We can face it!
In an ideal world the problem that some sites will break, if we were true to our bc break promise shouldn't even exist. We live in a real world that sometimes things go bad but we can face it (eg more testing, educate backup, next release creation to address the problem we introduced).
And you know what, the ones that don't want these problems will just opt out from this feature and let their administrators take care! The ones who want it will prevent their site from being hacked 4-5 times a year and might once per 2years when something goes bad, their site won't do something for 2 days...
I think the comparison's result is obvious!

Imo the problems which should be adressed about this issue is not where the certain advantages outweigh the probable disadvantages but how to face the practical technical implementation problems and how to communicate it...

Ps. I thought of a way to prove myself for the claim of majority of joomla sites :) Can @mbabker do his magic with the stats we have and present us some numbers?

Ps2. the autoupdate should't be only limited for 3.8.1 to 3.8.2 to 3.8.x only but should go to 3.9.0 too!
(i let this for the end so the brave of you that still read dont abandon me earlier :) ) There is no point in the autoupdating system if let's say my system autoupdate today to 3.8.5 and in 15 days we release 3.9.1 which adress security issues for the whole 3 branch... Ok there is a point cause all the sites are gonna be safer for the 3.8 release but still our bc policy dictates that 3.9 should be one click upgrade too. Why not do better than wp here? :) Ok i pushed my luck here. I believe this point is debeatable :) However the other option that we backport the 3.9.1 security to 3.8.5.1 i think will make joomla release workflow much harder maybe harder that just auto update to 3.9:)
Ok i pushed my luck here. I believe this point is debeatable :)

Ps3 sorry for the long post and bad english.!

3.9 release blog---> We are glad to introduce this long awaited great feature bla bla bla ....
... for technical reasons in order for joomla sites to be benefited from this, the update to 3.9 must be done manually. From 3.9 and going forward the ones that opt in for this feature will be able to auto update.

In my example above i have used 3.0.0 as this version is already released but this is also true if we start now. At the beginning you just need a few packages but with the time this is growing.
As for this problem it does not matter where we start but more how long do we support any released version. 3.0.0 as example was released in 2012 so lets think of 4.0 and 3.9 are going to be released this year. Than for 3.x.x would come the security only phase of two years. If you do the maths this is 8 years of major version support. Where you need to take care and update if needed all versions in between. This is going to be a big number of releases to support and to take care of. I don't see where the people are comming from who can handle that.

Ps2. the autoupdate should't be only limited for 3.8.1 to 3.8.2 to 3.8.x only but should go to 3.9.0 too!

Please explain that a bit more. Sure you can update from 3.8.0.5 (sec only update) to 3.9.0. But that would be no autoupdate as this is also including features and bug fixes etc. that conflict with 3rd Partys. ;)

However the other option that we backport the 3.9.1 security to 3.8.5.1 i think will make joomla release workflow much harder maybe harder that just auto update to 3.9:)

I get your point. That b/c policy thing is true for the core but we have no control over the extensions that needs to be updated first. Another point is If you install a 3.0.0 version of joomla and this get autoupdated to the patch level of 3.9.0 this means also that the extensions installed on that site (including the core ones, expect changes for security) are still the same as 3.0.0 so you can't just auto update to whatever stable version is out now as that would break any extension not compatible with that version you update to.

PS. I totally get your point why you (and others) want to have automatic updates. But for now it looks for me we try to do the second step (Automatic updates) before the first one (signing updates).
And also the first step include changes in how we distribute releases. (for sure this is mostly hidden from the public) :)

I probably didn't explain myself well.
There are two topics here.

1. The "easy" one: Let's say we start the autoupdate feature at 3.9 . The installations which are <3.9 obviously won't be able to use it. The autoupdate path will be incemental and one way. For example someone that is 3.9 and didn't opt in for this feature at start and we are already at 3.9.5 and then instead of manually updating prefers to tick the autoupdate then automatically his system could go 3.9->3.9.1->3.9.2 ...->current. So only one path for autoupdates to reduce the extra paths-burden. Ofc the usual builds you guys do for manual updates should be possible to be downloaden 3.9.x to 3.9.5.zip for example.
   You plobably thought i was suggesting the 3.9.1.2 release for security patches. I did not. I think that this would add excessive burden for the release leaders. I suggest that the autoupdate should make 3.9.1->3.9.2 for example.
   As far as extensions are concerned there are two strategies for them. 1) Hey joomla fellows we support the new exciting joomla autoupdate feature and we make certain that our extensions will be ok. 2)Hey joomla fellows the new autoupdate feature sucks. If you want to use our perfect extension plz make certain that you don't opt in in this feature or you probably will regret it.
   Either way there is a sollution. But the main gain remains a respected number of sites will be much safer!
   I wouldn't restrict this only to security patches. Why not performance and bug fixes? That's why i suggest autoupdate to the normal 3.9.x releases and not 3.9.x.x.

2. The second is whether to autoupdate from 4.0.9->4.1.1
   Well it seems much more daunting but in principles it shouldn't be much more different than the first case. The bc promise still exists! It can be a release that adds features but that doesn't mean it break old ones... I guess the scary part is that adding features means much more code so much more probabiblity for unintentional bc but that's it. In principles it shouldn't be different than the 4.0.8->4.0.9

Here's the problem. If you only do one autoupdate channel, and only do autoupdates using the same strategy we do now, then sure it might be a proper test of our B/C promise and strategies but it doesn't fulfill the real life needs of some site owners. Some site owners ONLY want autoupdates for security releases. We do not issue security only releases unless there are exceptional circumstances. So even autoupdating to those security only releases, if you are not on the release immediately before that then your update is pulling in other fixes not related to the security release.

The reason autoupdate works for WordPress is:

• They only stay in the current x.y release line (i.e. we have a client site still on a WordPress 4.8 release, it will never autoupdate to 4.9)
• Their "patch" releases only contain critical bug fixes and security fixes (and the bug fixes generally stop after the .1 or .2 releases, meaning it gets to a point where they are only security fixes)
• They are still backporting to every branch that supports autoupdates, so even those folks who have never hit the manual update button once getting onto 3.7 are still getting security patches

So, though they have done something in making autoupdates available, their release strategy for a non .0 release is very conservative which makes applying those autoupdates mildly safe. But they do not push you onto feature releases, and I don't think we can blindly do that either.

Some site owners ONLY want autoupdates for security releases.
These site owners also don't care for performance gains or bug fixes if it just works. Ok i can accept that there are these site owners but they should be able to afford backup mechanisms or just not tick the autoupdate feature!
I can understand that the perfect thing would be backporting security releases to previous versions, or making only 3.x.x.x releases and keep anyone safe... But from my understanding this needs lots of complexity -> manpower -> money. And it's in my understanding that this cannot be done in joomla.

However! Since this cannot be done in joomla cause lack of resources let's do the next close good thing that can be done! Let's make this linear autoupgrade come true for the ones that want it!!!
Nah... That's a lie... It's not only for those that want it! It's for joomla as a whole! Tell me how joomla looks now that only 25% is in j3.8 (i know it's not accurate but still indicative) ? Isn't it sad that now you see 3.1 or 3.2 sites (full of malware) i guess?

There will be an answer for all those that want eg only security patches etc. Joomla has made this autoupdate feature with these characteristics. Obviously, in your case you don't need it so don't opt-in!
And you know what, the ones that opt in will know that there is a posibility that something might not work as it should and they have to take the risk...

Risk? What risk? Isn 't it the same when someone presses the button update now? Everyone today when presses this button knows that 3.8 to 3.9 is one click update. However he still knows that there is a small posibility that this might go wrong... So by not implementing the autoupdate feature how do we win?
By "abusing" the late reactions of most sites owner! In an ideal world evereyone would update the same date and everyone would see this break in his site...

Don't misunderstand me! I don 't mean we should act irresponsibly. We should check better our releases for bc breaks (imo this feature will help towards this direction a lot!). Autobackup mechanisms (with extensions like akkeba?) will surface and probably become necessary! However, just because we cannot do the perfect security-only-patch-for-every-edition-since-10-years autoupdate mechanism don't drop the one that we can do!

ps And i thought i wasn't really talkative as a person :)

Personally I have to draw a line. A first version of any autoupdate system cannot autoupdate to feature releases (any .0). That is too high risk for a first cut of the system, and in general too high risk when you start accounting for the combinations of extensions running on sites (and let's be frank here, how many times have we seen template providers tell users to wait weeks to upgrade their sites until a new version of the template is released compatible with the new Joomla release?). Which means the effort needs to focus on patch releases at first. Which still gets back into the debate about whether there should be an upgrade path that is truly security only (means more backport/patching) or if the autoupdates only pull the latest stable on the minor branch you're on.

Risk? What risk? Isn 't it the same when someone presses the button update now?

One action is done by the user and one action is done by us basically. There's a liability aspect as play in that case, especially in lawsuit happy cultures who would find any loophole to try and sue Joomla/OSM because an autoupdate took down their site.

The whole reason for this item is to have the discussion on the hurdles and concerns with implementing it. Not to be a sounding board for everyone who dislikes the idea to shoot it down ?

I am generally in favour of autoupdates
I am not in favour of having autoupdates ONLY for security patches
I dont care about anyone who has "hacked" the core code

I guess one step at a time is wise :) However, then a very practical problem arises. Let's say 3.5.4 has been autoupdated. In 30 days 3.6 is out and in 5 days a security comes up. The whole auto updated mechanism has become useless and doesn't serve the very first purpose for its existence...
As for the law implications, we see wordpress to auto update without asking, windows 10 the same and many pcs with death screens and they would suit us when it will be opt in after reading the terns ;)

The whole reason for this item is to have the discussion on the hurdles and concerns with implementing it. Not to be a sounding board for everyone who dislikes the idea to shoot it down ?

I am really really glad for this. After seeing this i believe it is for the experts to discuss the technical staff :)

Brian sums up in 3 sentences what i tried in 3-5 very long posts ?

Well, my 2 cents, i believe it can be a very great feature.
And i guess that if it could be an "opt-in" feature, we can cover the majority of situations.

If @mbabker the line you drew cannot be removed then my proposal for the problem i described two posts above is:
A "contract - promise" with the joomla community that we will backport any security release to the previous version for a "grace period of one month" (or two). So the template-extension developers and site builders can have that month to ensure that all works ok with the new version.
Example 4.1.5 is last version of 4.1. Today 4.2.1 comes out. after 2 weeks 4.2.2 comes out with bug fixes. After 1 week a security release comes with 4.2.3. This security patch is backported to 4.1.6 (or 4.1.5.2) since only 20 days passed. In another 15 dayS another security patch comes with 4.2.4. We don't backport it since the grace perios has passed. By now all should have updated to 4.2.
The grace period can be 15 days 1month or whatever you like:)

we can only have one official version. otherwise it makes no sense

I know this... however if what we state was actually believed too then there wouldn't be these cases Michael described that "force" us not to implement the autoupdate between minor versions. So if we don't implement between versions and don't do something like the backporting the problem remains

However, then a very practical problem arises. Let's say 3.5.4 has been autoupdated. In 30 days 3.6 is out and in 5 days a security comes up. The whole auto updated mechanism has become useless and doesn't serve the very first purpose for its existence...

Feature releases, regardless of how well we adhere to a B/C policy, are high risk. They introduce new features to the API and no matter how well we code things site do break because of them.

3.5 added methods to the JForm and JFormField classes. It broke extensions overloading those classes because the new methods could not be found.

The last couple releases have added methods to JHtml helper classes. Because those classes were overloaded, it broke sites.

So even though introducing new methods to classes is not a B/C break, for all practical purposes it does break things for one reason or another.

That's one of my main arguments against supporting automatic minor updates right now. We might be B/C per SemVer and the practical testing we can do, but it is still a high risk action that causes breakage by no fault of most people. That's why initially we need to stay focused at patch releases, which are by design much smaller scopes of change and should not be changing APIs keeping the potential of breaks minimal.

I think auto updates are a must have feature. It will increase security and thus the overall reputation of Joomla.
I think that you should be able to set one of the following settings
( ) Disable auto update
( ) Auto update security patches only
( ) Auto update all minor and patch releases

for the signed package validation i've opened a new [RFC] #19792

My two cents are as follows:

A configuration setting Auto Updates On / Off (off by default

In the Joomla update system a roll back feature.

Options are what has always made Joomla great!

roll back is incredibly complex and impossible without a full file and database backup. there is a reason that there is no roll back on update available anywhere in the joomla world

Yeah, the rollback feature would be absolutely horrendous and absolutely not practical. Akeeba goes a hell of a long way towards making this not necessary, however if you relied on that you'd still need some way of notifying the admin that the update didn't go as planned and they need to deal with it as their site's offline.

there is a reason why akeeba doesnt have a rollback ;)

No, but it could be treated as a path to rolling back in the event of the update not going as planned, assuming of course that it was installed. It's far from ideal of course and I'm not advocating it. Just slinging it out there.

the only real way to rollback is to rm -rf and then restore from backup

this is why i'm scaried about automatic updates, it's quite hard to find people adopting the basic backup 3-2-1 rule to begin with

There are bigger fish to fry than trying to implement this feature. Like getting 4.0 stable enough to even have a beta.