[#141] - Issues with data processing in web hooks
- Closed
- 21 Aug 2013
- Medium
- Build: master
- # 141
It looks as though we have some issues with our data processing in the web hooks at the moment.
Issue 1
For my last comment on #77 we have the following rows:
INSERT INTO `#__activities` (`activities_id`, `gh_comment_id`, `issue_number`, `project_id`, `user`, `event`, `text`, `text_raw`, `created_date`) VALUES
(54, NULL, 77, 2, '', 'open', NULL, NULL, '2013-01-24 05:55:40'),
(55, 0, 77, 2, 'davidhurley', 'open', '', '', '2013-01-24 05:55:40'),
(56, 22907968, 77, 2, 'mbabker', 'comment', '<p>Welcome back to issue 77 :-D</p>\n\n<p>As we''ve progressed on the data processing, one thing we need to do is add in other activity logging. Joomlacode logs changes to the title, category, and status (maybe more, I only took a look in a couple of tracker items). The discussion at this point should focus on what we want to log then act accordingly.</p>', 'Welcome back to issue 77 :-D\r\n\r\nAs we''ve progressed on the data processing, one thing we need to do is add in other activity logging. Joomlacode logs changes to the title, category, and status (maybe more, I only took a look in a couple of tracker items). The discussion at this point should focus on what we want to log then act accordingly.', '2013-08-19 22:00:39');The first two rows are from adding the base issue to the database (the comment hook adds the issue if it isn't already present) and is double processing somewhere with a bad dataset. The comment dataset is good.
Issue 2
On the app at http://issues.joomla.org/tracker/jtracker/77 the original issue text is incorrect. This is an issue somewhere in the processing of the issue, here's the row for that issue:
INSERT INTO `#__issues` (`id`, `issue_number`, `foreign_number`, `project_id`, `title`, `description`, `description_raw`, `priority`, `status`, `opened_date`, `opened_by`, `closed_date`, `closed_by`, `closed_sha`, `modified_date`, `modified_by`, `rel_number`, `rel_type`, `has_code`, `labels`, `vote_id`, `build`, `tests`, `easy`) VALUES
(20, 77, NULL, 2, 'Event Types on Activities Table', '<p>Welcome back to issue 77 :-D</p>\n\n<p>As we''ve progressed on the data processing, one thing we need to do is add in other activity logging. Joomlacode logs changes to the title, category, and status (maybe more, I only took a look in a couple of tracker items). The discussion at this point should focus on what we want to log then act accordingly.</p>', 'Currently the event types is a VARCHAR field. It appears we have comment / open / close as data that is tracked by this activities table. Will there be other event types that are available. When looking at reporting etc, it''s nice to know all possible activities we''re tracking.', 3, 1, '2013-01-24 05:55:40', NULL, '0000-00-00 00:00:00', NULL, NULL, '2013-08-19 22:00:39', NULL, NULL, NULL, 0, '', NULL, '', 0, 0);The raw text is the issue text but the Markdown parsed data is from the comment that triggered the insertIssue() method in the comment hook.
Unless something has changed with the comment payload, the issue data should always be present (see https://gist.github.com/mbabker/6275111 for an older payload I have cached). I do agree we should get the full comment history in the app when it's being introduced in.
As for doing a full import, that would solve the insertion issue, but then we're only masking an issue .
Well, half of this is cleared up now.
As @elkuku pointed out in #77 the IssuesTable class has code to handle the activities. There's also code in AbstractHookController that's still being triggered, which is causing the duplicated open activity. So, there's some cleanup that needs to happen between those two classes to remove the duplicated effort and ensure we're pushing the correct data into the table.
So this is working now ?
Agree, we have to clean up that code.
You may blame the J!Tracker Application for transmitting this comment.
Issue 2 here is good. Just need to remove the duplicated code to fully
close this.
On Tuesday, August 20, 2013, Nikolai Plath wrote:
So this is working now ?
Agree, we have to clean up that code.
*You may blame the J!Tracker Applicationhttps://github.com/joomla/jissuesfor transmitting this comment.
*—
Reply to this email directly or view it on GitHub#141 (comment)
.
With any luck, that last commit closes this one out. Guess we'll wait it out and see.
Getting there. joomla/joomla-cms#1419 was just added to i.j.o but there's still a couple oddities:
INSERT INTO `#__activities` (`activities_id`, `gh_comment_id`, `issue_number`, `project_id`, `user`, `event`, `text`, `text_raw`, `created_date`) VALUES
(144, NULL, 1419, 1, '', 'open', NULL, NULL, '2013-07-06 02:21:11');The open activity is still missing the user data here. All of the text and comments are pulled in and in the right places, so that's a huge plus (compared to when I opened this issue).
-
I shall consider this closed after that last commit and testing it.
I think the problem may be only if a comment is received without an associated issue. I think this should better result in an error rather than trying to create a new issue.
The correct behavior would be to 1. Import the issue and 2. Import any previous comments before adding the comment from the hook to the database.
But I think that if you run a proper setup, importing all issues and issue comments using the CLI script, this should never happen.
?