I don't like it much, we still have the plugin in our dispatcher, can't we really just not load it? or only load it when it is really triggered based on getEventListners?

we still have the plugin in our dispatcher

Nope, it not there, check it ;)
If you do

dump($dispatcher->getListeners());

You will see that dump() does not contain Dummy Plugins.

can't we really just not load it?

We could return false for $app->bootPlugin() but intrface require to return PluginInterface.

or only load it when it is really triggered based on getEventListners?

First I thought it is a good idea, now after looking around, I think this will not help much.
You see, we still need to query all plugins from DB, and then check them on each Event trigger, whether they are booted or not.
This require rewriting our dispatcher, or have an extra abstraction on top of existing dispatcher, that complicates the logic.
I call this: much hassle, small profit :)

This PR just an idea, that already can work without much investment.
I am not insist on it, maybe we can find some better solution.

Addittionaly I would disable onBeforeExtensionBoot and onAfterExtensionBoot events for Plugins. But that another topic.

Thanks for caring about this issue. The big problem with this approach is that service providers should not contain any logic. Their job is just to provide the services. The logic should be somewhere else. I mean we have a client flag in the db record, perhaps you can use that one much earlier in the life cycle.