Next step:
This now generates a comment on GitHub when a PR that has been tested receives new commits, mentioning the testers so they get notified if they decided to subscribe to the PR updates on GitHub (as by default).

I have set up a test site now that contains the code proposed in this PR.

Note that we still have a quirk with accessing projects with full URL.
e.g.: https://trackertest-elkuku.rhcloud.com/tracker/jtests/86
might come up with an error if you're not logged in. Fixed in #689

Feel free to test
https://trackertest-elkuku.rhcloud.com/tracker/jtests

There is also a cron running that syncs every 30 minutes from GitHub.

@mbabker could you give me some infos about the hooks setup for the CMS so I can play a bit with the test project?

http://cl.ly/image/1Q0P2k2L403i
http://cl.ly/image/3y3f2f2O2D1L
http://cl.ly/image/0g0X471K1T30

Should actually be the same config for all repos pushing to the app right now.

Thanks, that was helpful

Testing site is not loading :(

Well the server is just a test server that goes down if it does not receive requests. Could you just reload in case you got a 503 please?
Sorry but that's the best I got available currently ;(

I am not sure about

the test options have been removed from the comment box at the bottom.

Need opinion from JBS guys @zero-24

PR works, but I think UI should be improved.

1. When I hit "Submit test result button" nothing happens, but actually it does. Need some kind of spinner to display while the process is going on the background. Otherwise a user will try to hit the button one more time.
2. "Test successfully added" text should be more highlighted and may be comment box should be cleared.

the test options have been removed from the comment box at the bottom.

Can we than have a Test this button there?

Maybe we can use insted of --> (:white_check_mark:)

Maybe we should add the comment with the Projekt comment
This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/XXXX.

The message This PR has received new commits. Should be expanded to This PR has received new commits and tests are reseted please retest. (maybe a native speaker can have a look here?)

Please make sure the alter test result box don't send a message.

@b2z

the test options have been removed from the comment box at the bottom.

This has been done to avoid "duplicated elements" on the page. It would have been necessary to also add the SHA code from this PR to the comment box at the bottom (along with the JS processing...).
I believe the original request also was to add a comment option to the test field ;)

UI should be improved

I fully agree here on both of your points but actually nothing has changed since we had this same (bad) behavior before this PR.
I'd love to find a more general solution the we could use in our application for AJAX requests "waiting" and "result" behavior - could we think about this in another PR please?

@zero-24
The errors you mentioned should be fixed (and deployed) - Thanks!

Can we than have a Test this button there?

This is gonna be tricky - since the comment "field" is now a hidden div, it would have to be made visible and the page would have to be scrolled from the bottom to the top (using JS?) --- not sure if this works reliable especially on mobile devices...
Don't you think a red button at the top of the page stating Test this would be enough?

Maybe we can use

Implemented

Maybe we should add the comment with the Projekt comment

Since we had some discussions about the wording I guess this should be good
Implemented

The message This PR has received new commits. Should be expanded to This PR has received new commits and tests are reseted please retest. (maybe a native speaker can have a look here?)

I'd love to wait for the discussion by the native speakers when this PR is merged (please)

Please make sure the alter test result box don't send a message.

It shouldn't.

BTW: Could you (or @b2z) please test the "alter test" functionality? You should see the button by now.

One question here: Do you think that it would make sense to post a message to GitHub when an item gets tested (not altered) - the same but without the comment?

Thanks @all for testing

hmm it looks like that it don't work with the sha / "Please retest" message? See: https://trackertest-elkuku.rhcloud.com/tracker/jtests/92
The sha is also missed there.
And the human test result is not on github?

Do you think that it would make sense to post a message to GitHub when an item gets tested (not altered) - the same but without the comment?

If you test successful or unsuccessful for sure. I'm not sure with the Not tested button.

BTW: Could you (or @b2z) please test the "alter test" functionality? You should see the button by now.

Works.

Can we have something like the test successful etc. also on the edit view?

Usecase:

• A Pr gets one test
• I just test the PR
• I open the Ticket
• add my test
• comment that i have tested it
• go to edit
• set RTC
• comment that i set it RTC

if i can add my test also on the edit view so i don't need multible comments / steps that would be awesome ;)

hmm it looks like that it don't work with the sha / "Please retest" message?

OK this will give me another round of headache
Problem here is that the commits for a PR are not fetched if the PR gets added by a web hook...
This will require another 20+ lines of c&p from the CLI script where the commit history is fetched...
I'll give that some thinking later today and post an update

The sha is also missed there.

Same here

And the human test result is not on github?

This will start working after #695

I'll also do some thinking about the rest of your comments - thanks again for your input :)

OK some things have changed here:

1. Stuff should now work as expected (hopefully)
   Test: https://trackertest-elkuku.rhcloud.com/tracker/jtests/101
2. The test facility is also present in the issue edit view (as requested by @zero-24 - I hope this fits your use case )
3. A comment is now always created on GitHub when a test result is submitted on jissues. (as discussed here: #683 (comment))
   3a. The SHA where the test has been performed is added to the comment.

Please test again

I forgot.. @b2z regarding your comment here #683 (comment):

While adding the comment box the div containing the status messages slipped away from the submit button...
It is now back right above the submit button with its submitting... and submitted message.
I hope we'll find a better solution one of these days

The test facility is also present in the issue edit view (as requested by @zero-24 - I hope this fits your use case )

Just found that if we alter or submit a test there the toolbar is still "User tests: Successful: 0 Unsuccessful: 0"

A comment is now always created on GitHub when a test result is submitted on jissues. (as discussed here: #683 (comment))

+1 for successful & unsuccessful. But also for not tested maybe we should name this reset test or something like that? Or leave it alone? (=> no message)

As the Message I have not tested this item don't helps ;)

3a. The SHA where the test has been performed is added to the comment.

hmm looks like it don't work every time?

https://trackertest-elkuku.rhcloud.com/tracker/jtests/101#event-737

Just found that if we alter or submit a test there the toolbar is still "User tests: Successful: 0 Unsuccessful: 0"

and

hmm looks like it don't work every time?

I hope both issues are fixed with the latest commit.
Please re-test with a new PR ;)

About the wording of the "not tested" comment: I think this should be a somewhat exceptional action, so maybe we can rely on the (human) user to add a clarifying comment?
I'm pretty sure some discussion(s) about the wording will follow as soon as this reaches the general public

Works great I have just noticed on very minimal see: https://trackertest-elkuku.rhcloud.com/tracker/jtests/103

if i reset the test (or alter a test) we have a activity entry.
if i add a new commit the tests get resettet without activity entry. Can we have such activity entry there? Maybe as user the Tracker bot?

@elkuku

Yesterday i got respons on UI/UX by @crystalenka

She had some thoughts:
• tooltip next to the SHA code is good, but probably should be revised before publishing ;)
• This is a style note, but it will be easier to consume the info if each 'event row' had a little bit more space in between. Right now it's all very close together so it seems more overwhelming than it is.
• I'd make the 'test this' button a little bit larger and give it margin so it's not running up against the edge of the container.

And another point is the "test this" Button should not be red maybe blue.

Thanks @crystalenka !

Some more adjustments based on feedback - thanks @crystalenka

The button is now blue, moved to the left, adding some margin. I am not sure about the size, since it is also displayed along with another button if the user has edit rights like this:

The "standard" user view would be like this:

I have also removed the tool tip and the dummy text. I searched a bit about some more or less easy to consume information about the topic: "The git SHA" and found this:

• https://schacon.github.io/gitbook/1_the_git_object_model.html
• https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
• http://blog.thoughtram.io/git/2014/11/18/the-anatomy-of-a-git-commit.html

Not sure which is the most easy to consume and/or the less technical - maybe we can get something up on the docs - so I just picked the first from this list and added a link next to the displayed SHA - to scare away the curious

@elkuku
I like the button on the left - I was mostly talking about size because it was on the right and not as prominent. Moving it to the left solves this problem in a different way. :)

So #696 was merged and this need to be updated ;)

I've tested this PR :) The only problem is the initial SHA value - it is empty.

The thing is, that this might work reliable only for "new" created PRs...

I tested both web hook and (local) CLI import and everything looks just fine: http://jtracker.local/tracker/jtests/104

I guess the safest thing would be to run the CLI script with the --force option for the "old" issues that might receive tests to ensure that commit data is present.

@mbabker this one will need DB update before deploy. Thanks!

@mbabker can you give me 3 days to update the docs bevor you deploy this? So i can update it with the new way to test and alter the test result?

Too late, I already rolled it and we don't exactly have a rollback system since we lack things like version tagging or database migrations.

ok no problem ;) It is my problem that i have not done it bevor.

B/C break

B/C break

We can break as often was we have no version 1.0.0 tag as far as i know ;) all 0.X.X can break B/C as they have not official released ;)

1.0.0 will never be released

Looks like we have a "real" regression. #708

Let me try running the pull with that --force option and see what happens. Just know this will probably take a while.

Thanks. Is the job now ready? It looks like that is work now ????
joomla/joomla-cms#7908 (comment)

I will try it later ????

Works now for me see: http://issues.joomla.org/tracker/joomla-cms/7974 Thanks @mbabker

@mbabker What was that? Files were not update properly? Just asking

No, I had to run the CLI script with the --force option to update every single issue with the needed data.

ah ok, that's the note made by @elkuku

I guess the safest thing would be to run the CLI script with the --force option for the "old" issues that might receive tests to ensure that commit data is present.

Missed it :(

Sorry I am somewhat "off duty" for the next couple of days - traffic accident - but I'm alive and typing :)

Glad that the thingy is working so far, those issues were somewhat expected... sorry @mbabker for the extra work ;(

You can buy the bots a later