Steps to reproduce the issue

create a bilingual site, like EN and NL
create a menu item for each language, associate them
set the Language Filter to "Remove URL Language Code" No
switching the language in the browser shows urls: domain/nl/menu-title and domain/en/menu-title
no problem so far

set the Language Filter to "Remove URL Language Code" Yes

Expected result

switching the language in the browser shows urls: domain/menu-title and domain/en/menu-title

Actual result

switching the language from the Dutch domain/menu-title to English works: domain/en/menu-title,
switching back to Dutch creates url domain/nl/menu-title, not domain/menu-title
This causes a 404 error

System information (as much as possible)

Joomla! version 4.0.6
PHP Version 7.4.27
Web Server Apache/2

Additional comments

In Joomla! 3 this used to work