Custom fields allows you to group fields, but not assign them to a category at once. This ok if you assign the fields to a category every time, but it's cumbersome and error prone.
I suggest we add a 'Group selection' to the Categories. So when you edit a category you can select a Custom Field Group which applies the fields to this category.
This means it's a single point of organising the fields and assigning them to a category.
Imagine, you have create a category and setup everything perfectly. You now want to add another category but with the same fields, you need to do a batch change or edit the fields individually. This way when you create the category you select the group and it's automatically applied.
As we have a category in a custom field, it would probably make more sense to have a category field in the field groups?