Steps to reproduce the issue

Form:
Create main form with subform; Add Modal window field (custom) to subform.
Backend:
Click add subform. Click "select" on modal field

Expected result

• open modal window
• load iframe, display list of items

Actual result

• open modal window
• empty content

System information (as much as possible)

PHP 7.2.9 on IIS 8.5
Joomla 3.9.10

Additional comments

Found the possible solution in /layouts/joomla/modal/main.php, Line 86:
$script[] = " $('#" . $selector . "').on('show.bs.modal', function() {";

-->improper jQuery call for JS-added elements, so it will not be called on subform rows; should be:
$script[] = " $(this).on('show.bs.modal', '#" . $selector . "', function() {";

This solutions is almost working for me, just needed to fix another problem on line 120:
$script[] = " padding = document.getElementById('" . $selector . "').offsetTop,";

This throws an error document.getElementById() is null; I changed this code to the jQuery equialent:
$script[] = " padding = $('#" . $selector . "').offsetTop,";

Now its perfect working for me. Maybe this simple changes can be done for next release.