You don't want to totally remove permissions from the component since those provide the fall backs that are used in absence of specific settings. Ideally you only use the item level ones in special cases, similar to the article level permissions in content.

Also, please add a Feature Tracker on joomlacode an reference back here:
http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemBrowse&tracker_id=8549

Forgot:
You are deleting in this patch a new string just committed:
COM_MENUS_ITEM_FIELD_HIDE_UNASSIGNED="Hide Unassigned Modules"

I didn't remove the component permissions. I just moved them to the access.xml file along with the menu type and menu item permissions. Also, I see now in this patch that I didn't use the newest files when I made my changes. I'll close this request, fix it, create a feature tracker and then submit a new pull request. Thanks!

You shouldn't need to move the access.xml from where it was.

When I started, com_menus didn't have an access.xml. I created it and then moved the component permissions to the new access.xml from config.xml so it would look more like com_content. Looking at the commit, I don't think the access.xml I created got added. Sorry for the confusion, I'm a bit new to Git.

Oh I see it was just using the default.