Steps to reproduce the issue

Have your site setup in http://localhost/~/whatever/
this will make the base64 encoded return urls start with aHR0cDovL2xvY2FsaG9zdC9+
Create a category list menu item
log in frontend with admin. go to that menu item. Click on new button to create a new article. Then, click on cancel, or fill the form and submit

Expected result

after cancelling/creation, you should be redirected to category list menu item

Actual result

you are redirected to home page, because of 2 problems:

• there is a '+' in the url, it will be interpreted as a space, because it's not url escaped. So, in fact, when appending the base 64 encoded url, it should be first passed in urlencode().
• The second problem is that the article.add action will redirect to the actual edit page using a linked parsed by JRoute::_(). This function will call UriHelper::parse_url() function to parse the paramters. This function runs urldecode, so it makes the first point moot... the urlencoded base64 value will get urldecoded in the generated link.

So, finally, when we cancel or submit, the controller will fail when parsing the return url, and redirect to home page as a fallback.

System information (as much as possible)

joomla 3.8.3, but happens to previous versions as well

Additional comments

It's probably not a good idea to have the return url in get requests anyway, but still should be fixed.
Another solution for cleaner and safer redirect would be to store the actual return url in a session var, and use an uniqid as the key to pass to get/post request and recover the value:
$returnToken = uniqid();
JFactory::getApplication()->setUserState($returnToken, $realUrl);
$append .= '&return=' . $returnToken;