It won't hurt anything. The problem is after you do a composer update more often than not the gitignored files are going to be in your local filesystem, so when Composer generates the autoloaders because the files exist they'll be included in the maps. Short of writing a script to execute at the pre-autoload-dump event to clean this up, you're not going to consistently fix this.
Just for reference, it's a little better with the 4.0 build script because that explicitly runs composer install --no-dev --no-autoloader then cleans out non-shipped files before running composer dump-autoload --no-dev --optimize, but it still relies on regular review of the Composer installation and updating the script's clean_checkout() function.
@mbabker the phpmailer update prevent the use of urls in addAttachement, even this maybe would be a b/c break for us. I'm not sure show this function is exposed in joomla.
Is it normal that we have the test classes in the autoloader?
e.x. Joomla\Session\Tests\StorageCase