From what I found out while debugging it's all fine until this line in JRouteSite::parseSefRoute()
https://github.com/joomla/joomla-cms/blob/staging/libraries/cms/router/site.php#L403
Here, the active component ($vars['option']) from the request (com_contact) gets replaced with the component from the active menuitem (com_content).
A bit further down (https://github.com/joomla/joomla-cms/blob/staging/libraries/cms/router/site.php#L442) it then takes the query params and overwites $vars with it.

So what is supposed to win here? Because you're passing a URL with a valid menu item configuration then also passing query string parameters. Any combination of any valid parameter (view, layout, and option being the ones most prone to being damaging) will do this, and honestly changing it would probably be a borderline B/C issue.

Actually I thought those type of links used to work in earlier versions.
But even if I'm wrong here, I don't see why those arguments should be added to JInput if the menuitem isn't for the requested component. Looks like a bug to me.

Where would you see a B/C issue here? Do you think there is code which expects to "inherit" menuitem query arguments from a different component?

I haven't tested it yet beyond the URL in my example but imho something like this should fix it without causing damage: staging...Bakual:FixRouter
Basically just don't overwrite the already set component when parsing the URL and don't replace the $vars when the component doesn't match.
But need to test it more before I can do a PR with it.

You're still having to make an implementation decision either way; does a menu item's configuration always beat whatever is passed in the query string or does a query string always override whatever is set by something else? And honestly, I don't know if either behavior is right, and if there is a decision made, it needs to be consistent (option shouldn't be special).

You're still having to make an implementation decision either way; does a menu item's configuration always beat whatever is passed in the query string or does a query string always override whatever is set by something else?

That's already there, if a var is defined both in the request and in the menuitem query, then the request takes precedence. And of course it has to be that way or you couldn't go from a blog menuitem to the single article page. Request always overrules the menu.
The vars returned from the router are stored into JInput using the "def" method, meaning it's only set when not already present.

it needs to be consistent (option shouldn't be special).

The option is special in the Joomla universe because without it nothing will work. It's the single parameter that needs to be present in every request, implicit (in the menuitem) or explicit in the URL.

Doing a bit more testing I came to the conclusion that this type of link likely never happens in production (only when crafting URLs to test things).

If it's done from the root of the site (eg index.php?option=com_contact) it works, even if you set a different Itemid in the request. Both the implicit home Itemid and the explicit one aren't leaking.
Also the SEF counterpart /index.php/component/contact/?Itemid=102 does work.

So it's seems to be a special case that isn't taken care of because it shouldn't happen. I can live with that.

Closing until I see it happen in production (which I think should not be the case) ?