Steps to reproduce the issue

Emails with certain valid subjects are not sent, an exception Prohibited input ... is thrown.

Email subject should contain @ char and some extra non-domain accepted chars next, see the subject example below.

Execute the following code:

$app    = \JFactory::getApplication();
$mailer = \JFactory::getMailer();

$to      = 'test@test.com';
$subject = 'Test email for test@test.com {test}';
$body    = 'Test';

// PHP mail works.
mail($to, $subject, $body);

// Joomla way fails.
try
{
	$mailer->sendMail($app->get('mailfrom'), $app->get('fromname'), $to, $subject, $body, 1);
}
catch (\Exception $e)
{
	echo $e->getMessage();
	die;
}

Expected result

Two emails should be sent.

Actual result

Only the first email is sent via mail(), the second email throws an exception Prohibited input U+00000020.
We can see that the subject string is valid.

System information (as much as possible)

4.0 nightly

Additional comments

Joomla\CMS\Mail\Mail::setSubject() calls MailHelper::cleanLine()

MailHelper::cleanLine() for unknown reason calls PunycodeHelper::emailToPunycode() which is only intended to convert the email address to Punycode if required.

Joomla 3 uses the \idna_convert class which actually doesn't create issues with such subjects.
Joomla 4 started to use Algo26\IdnaConvert\ToIdn for Punycode conversion which throws an exception on invalid domain pattern the subject is validated as.

The main issue is that MailHelper::cleanLine() is used for non-email address checks, i.e. for a subject or for a sender name.
We have MailHelper::cleanSubject() and MailHelper::cleanAddress() but they are not used at all. Note that cleanAddress() actually doesn't perform Punycode conversion.

Logically, even the current Mail code in Joomla 3 is invalid, even though it allows sending emails with such subjects.

All MailHelper::cleanXXX() methods should be re-checked and used in the proper manner.