I spent some time looking into this and my conclusion is that category filtering is going to be borderline impossible to implement without a major refactoring of the way editor xtd buttons work. The reason is the following:

1. Article view triggers plugin to load the editor
2. Editor triggers the plugin to get the xtd buttons

Neither the xtd plugins (or the editor plugins) have access to the article information because the editor plugin doesn't care to an extend about anything other than the data it's rendering. As a result editor xtd buttons are also unaware. Without literally reloading the active object and grabbing it back from the database we don't actually have a way to access the category id (and no easy way to do it b/c because even if we did it in say tiny - no guarantee that 3rd party editors would pass in the required info.

Additional comments
I can somewhat bodge the subform only part - by forcing the value of the filter for that because I can't think of any component where we'd want subform fields rendered by default. But it's not exactly straightforward either - because that modal is literally the fields list view you get through the menu pages - you'd actually get the same fitler applied by default if you visited that page in the browser later on (this would also apply to the category thing even if we could get the active category id into the field plugin)