but at least we should publish more ALPHA the last one is from November 2017 https://github.com/joomla/joomla-cms/releases/tag/4.0.0-alpha2
working with 4.x staging only can become a really time consuming game (at least for me)

We need to be carefully with the merge button.

As release lead I gave my best to get others opinions and tests on a PR. Sometimes you have to break the rules to move forward, but that should be the execption. Our CI should never be in a state that a PR break it because this takes hours/days to fix it and after a while nobody has trust in our testing.

Merge in haste, repent at leisure

Longer response coming after coffee but the tl:dr is you cant run joomla the same way you run your own extension business

It is not about to not gather other opinions or haste merging. It is about should we have the same slow development strategy as we have for patch releases on staging?

I wouldn't say our strategy for patch releases is slow, we made a lot releases over the last year many more than we had 4 releases. The problem with the slow development for Joomla4 is that we have not enough people working on it.

This was not a critic, and I'm all in for that on staging. I think it has improved the stability of Joomla. But for a new major release things are different.

I dont see the difference. @mbabker explained far better than I could ever do in the CODEOWNERS pr why merging in haste without the approval of subject experts is a bad idea.

Suddenly merging the 100+ open PR will not bring the release of J4 any nearer.

All it will do is to create more issues on the tracker that need to be resolved as seen by the 140+ open issues. A large percentage of which were created as a result of the alpha release occurring before the ui was ready.

This just wastes the time of our limited resources as people test things in releases that should never have been merged (or were merged knowing that they were not ready but this was not communicated).

Spending the week with @mbabker we realised that there are very very few current contributors who have gone through the process of a major release (and not just one, we are such suckers for punishment we have done it many times).

We must learn from the lessons of the past and not repeat the mistakes. A rush to make a beta release at JAB (and I am sure thats what this policy change and the invisible J4 lead is aiming for) will be a disaster. We cannot afford to have any bad pr from our first truly major release since 1.6 and we will be judged by the quality of the beta just as much as by the final release. A bad beta may even kill the ability to have a successful real release.

You have to think beyond speed and look at all the issues affecting the success of a release

Just to be clear here, this issue is not about to blindly merge everything. It is about that we are aware in what for a development state Joomla 4 is right now and act accordingly.

If you build up something from scratch like a new template or media manager, then there will be a long time where it is not ready. But on some point you have to get it in to make the next step.

In an open source project it is hard to have a preview or alpha phase because at any time everybody can check it out and test it. I agree that we should have make clearer statements in the alpha release announcement and I'm sure we will do in the future.

@laoneo sorry I am a bit lost, what will you change?

I want to clarify the situation where we are with Joomla 4 actually. The comments in #20088 gave me the impression that people expect the same dev strategy as on the staging branch, where we do patch releases. It is just not realistic for an alpha phase of a new major version.

You cannot compare media manager and the template as they are being developed in their own branches.

And yes I do expect the same quality strategy as in the staging branch

349 commits since Alpha 2 last November.

Ideally there need to be more alphas released so more external testing can be done.

Why not merge what has currently been done in the template repo and get another alpha released? It will give people a good usable preview of the designs made

@alikon

working with 4.x staging only can become a really time consuming game (at least for me)

https://developer.joomla.org/nightly-builds.html ;) There is also the updateserver that you should be able to use. But this require a working product in the 4.0-dev branch.

The comments in #20088 gave me the impression that people expect the same dev strategy as on the staging branch

Yes. At least not a complete unknown strategy that we had until today.

Ideally there need to be more alphas released so more external testing can be done.

In the announcement please make clear what is ready (in the view of the sub project team like template, mm etc). So the external tester know where to look and fix bugs and where they don't need to invest the time into as it is not ready at all or is going to be rewritten in the next releases.

i'm aware of the nightly builds ect, but didn't change too much the whole picture

And yes I do expect the same quality strategy as in the staging branch

But then we have to do an announcement that it is unlikely to release Joomla 4 this year. I think (yes I have no proof) that people expect Joomla 4 this year.

merging untested code wont get us anywhere nearer a release.

merging untested code wont get us anywhere nearer a release

In alpha state, this is not true, otherwise we would not have a compatibility layer in 3.8 and namespaced extensions in the 4 branch. There must not be done quality testing on alphas as things can and will change, so it is a waste of resources.

It is not only about merging, it is about the philosophy. That you can't expect a fully functional new media manager or new template in an alpha release. I mean I do also test Joomla 4 regularly and do provide small patches which do fix little things. I'm not saying that this should not be done. But we should not treat a version which is in alpha state the same way as a patch release. That's from a project management point of view, suicide.

you are missing the point. It is not about a "fully functional media manager". It is about not merging something that hasnt been tested.

Before I merge a pr which is not RTC and approved, I do always test them, scan the the code for obvious bugs and do have a look if the strings are alpha ordered, even the vendor dir removal, where you could only test if "composer install" is working on your computer. So at least it has one test.

Obviously people have different definitions of alpha. I still think that making J4 branch public was not a good idea (unless we had a solid alpha, eg most of the API done). Now we fight over tests, in an alpha version, which should never ever be used for anything else other that preview purposes...
Pitty

I fully agree with @laoneo and @dgrammatiko

We should not treat dev/alphas the same way as we treat releases.

Lets take a look at the roadmap and current progress. Knowing that the same ~5-8 people are working on all of those features, we should not make their life harder than it already is.

Please note, that this is only my view of the current progress of j4!

I give up. If you think its acceptable to merge untested code (and one test does not = tested code) then go for it. There were very good reasons that the testing policy was introduced.

Knowing that the same ~5-8 people are working on all of those features, we should not make their lifes harder than it already is.

The problem is that only 5-8 people working on it. I understand that these people would be happy if the process is easier for them. But this will not fix the problem of only having 5-8 people working on it. You end up in a situation where pretty much all is broken, nobody besides the 5-8 people know why and what should work and what can be tested. So nobody will join the 5-8 people and helping with Joomla4. If I have to search a day to find out what I can do to help I don't have time left to help.

... and so many times people report issues just to get the response - dont worry mystery team x is dealing with this.

Which is the same as saying "go away we dont need your help"

The problem is that only 5-8 people working on it.

For me this is a different problem and its not directly related to the requirements of 2-manual-tests and a no-break-in-alpha philosophy. Even if we keep the current requirements, we also only have a handful of guys doing the manual tests. So those testing guys are currently just a part of the mistery team x.

At least for me this issue is only about making it easier to move forward in dev and alpha stages and using our limited resources efficient, which does not necessarily mean that everything will be broken or some "mistery team x" will do hidden magic behind the scenes.

Its hard to make progress and implement new features or new architecture when nothing is allowed to break in dev/alpha stage. Transparency is a key factor, which can be ensured using proper tools like project boards, design documents, team reports, automated test, pr descriptions and the issue tracker.

no one said anything about a no-break-in-alpha
all that has been said is about the change in the test policy

Personally, I'm more concerned when Allon opens a PR and George merges it without anyone's testing/review/feedback. That is the type of workflow that's going to get us in trouble.

No, I don't think the strictest sense of "two good tests before merge" needs to apply in 4.0 either. More often than not when I run composer update I'm just direct committing; most of the time that is our Framework packages and I know exactly what has changed between each run of that since I'm doing the vast majority of commits to those repositories myself and in effect how that work impacts this repository. No, that's not following policy, but considering the only CI failures for Framework 2.0 branches affecting the CMS are PostgreSQL 10.0 support, I think it's safe to say that updating decently-to-well tested code can be done in a transparent manner until we reach the beta milestone.

Now jumping through comments here then I have a release to get out the door.

Its hard to make progress and implement new features or new architecture when nothing is allowed to break in dev/alpha stage. Transparency is a key factor, which can be ensured using proper tools like project boards, design documents, team reports, pr descriptions and the issue tracker.

I tried. I think 3 or 4 people were really interested in using the project boards and at the time they got set up I was the only person with write access interested in maintaining them. The project boards are an awesome project management tool, but since Joomla has no resemblance of a project manager...

I still think that making J4 branch public was not a good idea (unless we had a solid alpha, eg most of the API done). Now we fight over tests, in an alpha version, which should never ever be used for anything else other that preview purposes...

That goes contradictory to an open development process and what I believe are part of the core values of this project.

But we should not treat a version which is in alpha state the same way as a patch release. That's from a project management point of view, suicide.

You're right that it needs to be understood that the current state of the 4.0 branch is not a stable one, but that also does not mean that quality assurance should slip. I'm pretty sure everyone would get pissed off if I committed something that completely broke the database layer, or the session layer, or made com_content unusable. It's fine that the branch is in an alpha state, it's not fine that something gets merged which results in the software being totally broken. It's not fine that the same 5-8 people are working on everything, and in some ways proactively excluding others from being able to even participate because they are doing work and getting it merged without any form of peer review or testing.

Spending the week with @mbabker we realised that there are very very few current contributors who have gone through the process of a major release (and not just one, we are such suckers for punishment we have done it many times).

In this thread, it is myself, Brian, and Robert who have been with the project long enough to have seen at least 2 major releases (3.0 and 4.0), in some ways we've seen three majors (add 1.6 to that list) but in different roles/perspectives than we have today. The vast majority of everyone else actively contributing to the project right now came on at a point after 3.0's release. This is not a knock on anyone's skills, longevity, or anything like that, just a simple statement of truth; most people have not been involved with core for a major release. Even those that were here for 3.0 will know that in the grand scheme of things 3.0 was about as "major" as 3.2 or 3.8, except that release did make some deliberate B/C breaks as a major allows to do (and people are still more pissed about the template changes than the PHP API changes if that says something), the last time anything was attempted at the scale of what 4.0 has reached was the 1.5 or 1.6 development cycles (because 4.0 has gone further than what 3.0 did, there is no issue with that necessarily).

We need to set realistic expectations. We need to set realistic timeframes. We need to set realistic release milestones. We need to communicate all of this. We need a release management team (steered by one person, but the amount of work required is too much for one person, especially when that one person is already the head of a project department encompassing 13 of 38 of the project's "legal" teams) to do some project management to keep things on track and ensure the non-code related aspects of the release are also being focused in on (who's working on all those PRs labeled needing documentation?).

Probably I didn't made my point clear enough so let me try to make it a bit clearer.

I am expecting that we only merge tested code in our repositories main branches (staging, 3.9, 4.0-dev). The requirements for each branch are different. While we are very strict with stuff for staging or better when we are preparing a stable release, we don't need to be so strict when we are preparing an alpha release. A merge can break things, then we need to fix it or revet it make the PR better and merge it again. In some situations it is hard to find two people who are able to test and I have seen PR with more than 2 successful test that were plain wrong. So this is not a black/white question and the normal way should be 2 successful test. It is the job of the release lead to make decisions and break the rules.

Its not just the change for the human tests policy. Its about a change in philosophy. Alphas are previews, where we early release new features for other devs to see where we are heading to. For end users to get a preview what is coming next. If we have to wait for two successfull tests when somebody fixes the front end module ordering ajax request in alpha phase, we do something wrong.

If we have to wait for two successfull tests when somebody fixes the front end module ordering ajax request in alpha phase, we do something wrong.

Didn't we do something wrong before as we broke ordering?

Software in an alpha development state is by definition not stable. That doesn't mean that we bypass QA/QC policies and just commit things and fix it later.

I'm going to be blunt. Half the problems we are having right now are caused by the fact that George is overworked, different people on different teams have different working strategies, and that there is no one person involved with Joomla today qualified to act as a project manager (and that is a skill that has been sorely lacking in the core contribution circles for years). We need to get everyone on the same page working from the same set of rules. Without that, this ship's just going to sink and pretty soon we'll be on J4v4.

Its not just the change for the human tests policy. Its about a change in philosophy.

And this change was decided where and communicated where - oh wait it wasnt communicated anywhere at all.

If we have to wait for two successful tests when somebody fixes the front end module ordering ajax request in alpha phase, we do something wrong.

No not at all. We wait for successful tests to ensure that the fix is a valid one and not another break caused by a lack of testing in the first place.

Alphas are previews, where we early release new features for other devs to see where we are heading to. For end users to get a preview what is coming next.

And if something is completely broken what impression does it give those people about Joomla?

And when they take the time to report something and are told not to bother with a fix because we know its broken and someone is already dealing with it

And this change was decided where and communicated where

Its not. Thats why I opened this issue in public and not raised it in a closed glip chat. We need to have this discussion. Some peoole want to do it one way and some the other. If we do not find an agreement it will turn out in a mess.

So was @zero-24 wrong #20088 (comment)

Because from the sound of it and the fact that you merged stuff without tests which is against the current policy the decision has already been made.

In an open source project it is hard to have a preview or alpha phase because at any time everybody can check it out and test it. I agree that we should have make clearer statements in the alpha release announcement and I'm sure we will do in the future.

That's like saying it's hard to have a development repository in an open source project because anyone can check it out and test it in a non-stable state. We can have non-stable code in a repository and milestoned non-stable releases, that doesn't mean the repository can be completely broken or that QA/review processes bypassed. Nobody is trying to stall 4.0 development more than it has of its own accord, everybody is concerned that the commits going into the branch are not being QA'd or reviewed because "it's alpha, we'll fix it later". That latter point is what is going to shoot us in the collective foot.

Lets take an example. In pr #19834 I wait for over a month for feedback, all I do is to keep it in sync. It's frustrating and blocks me to convert all the modules if accepted (my plan was to do that till JAB).

If we would have a dedicated testing team, architecture group and a QA where you can count on that we get the reviews, feedback and tests in a timely manner. Then we can do that like on staging.

Ups, wrong button.

If only we had a place that all these different teams could communicate what they are doing and what their progress was. One of the aims of the "new structure" was to remove silos.

At least the "official teams" could report here
https://volunteers.joomla.org/departments/production#reports

As for the "unofficial teams" like the admin template they have nowhere even though they asked to be made a team 6 months ago.

If we would have a dedicated testing team, architecture group and a QA where you can count on that we get the reviews, feedback and tests in a timely manner. Then we can do that like on staging.

If only we had a dedicated media manager team where you can count on that they will produce usable code after multiple sprints etc etc.

Do you realise how rude that is.

Lets take an example. In pr #19834 I wait for over a month for feedback, all I do is to keep it in sync. It's frustrating and blocks me to convert all the modules if accepted (my plan was to do that till JAB).

Sorry to put it this bluntly, but a month's delay is nothing. I started work on prepared statement support in June 2016, the API is still not in a state where I can change even one query in the CMS without breaking something because ext/pgsql doesn't support a parameter syntax that any other database driver does.

@mbabker, nobody says we should not wait for feedback and blindly merge. Its all about to lower down the expectation that every pr needs the same process as one for staging. We do hard architecture changes in J4, where stuff gets changed multiple times like the extension service stuff. It is hard to get feedback for and even harder tests.

@brianteeman I was refering to missing teams and structures and did not target specific people. Perhaps my wording was not correct for native English speakers. But please do not start attack people and keep the focus on the issue we try to solve. Thanks.

the way you wrote it, it was an attack thats what I was pointing out to you. I was not stating anything about any specific team. None of them are reporting.

So it basically all boils down to what Dimitris said about people's perception of an alpha.

Those who have taken the time to air their views on this matter should probably come to a consensus with the production leader (or whatever the title is these days) and set that in stone, in regards to testing, merging, and releases.

Maybe even get the opinion of your president.

We all want things to move forward. Architecture is hard, it's what, 3 of us working on it (and what happens to efforts that aren't the 3 of us working on something, they stall out)? UI stuff is in a state of limbo pending the latest template design being completed (because I'm not spending time on UI bugs in my core areas (to include patch tester and IFW) when everything's about to change again). I have no idea how to get involved with automated testing now that all the tests are decoupled from this repo, to me because I don't see the end state for that work I just see it as a road block and another excuse to keep committing untested code (and yes I'm pretty damn guilty of writing new architecture with zero tests right now, thankfully much of our testing architecture is more functional than unit and the work I'm doing gets validated by existing test coverage).

At the end of the day I'm more concerned about letting QA/QC slip because "it's alpha, we'll fix it in beta". That's not acceptable to me. How we balance that against the perceived expectations that come from working in the stable release channel (3.8) is what needs sorting out. And I think that's going to involve a mix of working out the issues identified in this thread and a shift in mentality from the top down as to how the repository is managed in general (i.e. the code owner stuff because we can't rely on 4 or 5 people to do everything they are doing today).

I like roberts suggestion:
4.0@dev PRs need 2 human tests. Release lead is allowed to break this rule when it makes sense, but issues need to be created.

Additionally we can continue to move bigger features to custom branches or public repos. In best case those repos have an issue tracker, custom readme or project board, so that non-team members can get an idea of what the team is working on.

custom branches are fine as long as they are not closed silos which dont get opened up until it is deemed to be perfect.

@brianteeman you do realise that most of the work for J4 was done in a closed silo by a few people, right?
Also from the time that J4 was published under Joomla repo the progress is painstakingly slow...

Just because the real effort that started J4 was in a closed silo doesn't mean it should stay that way until we have a releasable product. We've come far enough with all the ongoing work that there is no reason anything to get to a beta milestone should be happening in a private repository or a closed silo where interested parties can't even find the resources to be involved (basically playing into the "5-8 people are doing all the work" problem).

I still think there is a good line between 2 human tests and none. Some PR's simply dont require them at all. A prime example was a styling fix to the input icons, which I made months ago to the login form, and probably amounted to 10 lines of code being changes. Cant remember the PR number but think I provided a screenshot. I do not deem that as something that requires 2 human tests. The same applies for other small things....not everything, but some. So there's a line. That PR by the way was a sitting duck after about 1.5 months so I gave up and just closed it.

A.K.A people lose interest in submitting if their code isnt reviewed. Be it merged or not

You can put it down to George being overworked which I respect, but it's up to him to say "I need help" and have an assistant appointed to him. Nothing wrong with asking for help. I've mentioned this to him personally too.

As for the "closed silo", well it may not be the best solution, but I can fully understand why people would want to do this

@C-Lodder and we just had a really simple language string change that would have been merged until it was pointed out to be completely wrong. the closed silo isnt having any positive impact on the admin template

You can put it down to George being overworked which I respect, but it's up to him to say "I need help" and have an assistant appointed to him. Nothing wrong with asking for help. I've mentioned this to him personally too.

He needs it. I'm saying this as I'm getting talked into managing my fifth minor release branch. There is a reason when I left PLT the first time I was advocating heavily for release TEAMS, not singular LEADS (yes, that team would be led by someone but the efforts to coordinate even a minor feature release are too great for one person to do the vast majority of it and that person needs to delegate a lot of tasks, code and non-code (marketing, documentation, etc.) to other people, my most successful release was when I ran 3.3 end-to-end because I started that having my close circle with key players in various teams to ensure things were getting done in a coordinated effort).

Brian, people make mistakes. We're only human after all. Nothing is ever pitch perfect and things get missed. This is how we learn from mistakes. I'll be the first to admit that. I've comitted code, I've asked it to be merged and then after realised mistakes were made, where I hadnt taken all factors into consideration.

All I'm saying is, decide what the plan is, stick with it and enforce it, whatever the outcome is. To be honest I'm extremely suprised this is even being talked about now rather than before J4 started to be worked on.

You're in dire need of a project manager. It's actually worrying

Brian, people make mistakes. We're only human after all. Nothing is ever pitch perfect and things get missed.

Of course we make mistakes - thats why we have procedures in place to minify that

All I'm saying is, decide what the plan is, stick with it and enforce it, whatever the outcome is. To be honest I'm extremely suprised this is even being talked about now rather than before J4 started to be worked on.

Most of us thought this plan already existed. We're talking about it now because some people appear to have decided to change it.

You're in dire need of a project manager. It's actually worrying

^^ That.

In effect, part of the release lead's role is to be a project manager for at least their release (as in they really should be more hands off than on in the code or the busy body work, but we don't have the resources to make that work so George and I get a lot more involved than some would like for us to be if we were taking purely management roles). And in effect, the production department leadership should have some form of project management role for dev as a whole on the brand's code (CMS, Framework, extensions, anything going out with our name on it). We don't have anyone to fill that kind of role and even if we did I'm not sure anyone would want to do it or that anyone would actually give that role the "authority" it would need.

Most of us thought this plan already existed. We're talking about it now because some people appear to have decided to change it

So basically whoever is there to enforce it hasnt done a good job at doing so?

I guess, everybody made his point. Pr's with RTC will only be merged also for Joomla 4. Going to throw away the milestone paper I did for Joomla 4 then, which is actually in review by PLT. We will see how far we will get in 2018 then. Thanks everybody for the input, I learned a lot.

Next time we have an issue that's barely open 12 hours where people are discussing the job i do it would be nice to tag me in it so I can actually give my side?

Anyhow. The aim with the J4 branch is that we're going to start to slowly stabilize. Now appveyor is back under control I'm going to start for example stop merging PRs with failing tests. However there are still 3 major pieces of work to do:

1. Finishing the J4 template (taking place in this open repo: https://github.com/joomla/40-backend-template)
2. Finishing the ES6 Javascript rework (taking place in this open repo: https://github.com/joomla-projects/joomla-es6)
3. Finishing Component Service Locator work (happening in this repo with some PRs e.g. #20170 and #20176 whilst we try options)

I expect the first two to have made serious progress by JAB and I'm hopeful to have the third completed in the next week or so.

Other features will be considered but are not alpha blockers. I'm still not going to require all code reworks at this point to have two good tests. However increasingly I will especially as the major 3 features are reaching full adoption (for example once the component service locator is done i'm pretty happy with the state of the PHP reworks we've done in j4 and i'll start to be tighter with what I'm merging).

TL/DR By the time we get to the later alphas/early betas we'll 100% go back to requiring two good tests + code review as per the j3 branch. Right this second we don't and won't for everything but will apply best judgement which will increasingly be tightened as our major 3 blockers near completion.

You can put it down to George being overworked which I respect, but it's up to him to say "I need help" and have an assistant appointed to him. Nothing wrong with asking for help. I've mentioned this to him personally too.

Me giving allon merge rights is part of me trying to rectify parts of this. unfortunately in the short term it means we're not being totally consistent with things whilst we get in sync.

Just to finally supplement this doesn't mean people shouldn't test!! I will not merge something that has a failing human test. it just means sometimes additionally i will choose to accelerate features without tests/with a single good test whilst we are in alpha

1. @wilsonge you havent responded when you have been tagged on multiple issues so maybe people have given up - i know I have.
2. removing the need for 2 good tests and review before merging is creating more work not less but if the project lead does it then I have less of an issue than just any of the rest of the people who have merge rights (including myself)
3. as for the state of the three major pieces you outline. You really have lost contact with them (or at least the template repo) which is no where near the proposal outlined in november. mainly due to lack of direction and leadership and nothing to do with a desire to make it happen by the one person actively contributing

I'm going to ask this loaded question then, since there seems to be so many interested parties here. I've been working on ensuring the database API can support parameterized queries for 21 months, and there is one final roadblock to making that possible - joomla-framework/database#112 - is that a focus point for anyone other than myself or can I stop worrying about the feature because it's such a low priority to everyone (or rather, everyone wants the feature but few are willing to do the work to make it happen)?