You have to load the field in the xml concerned. Example:

<fieldset name="request"
            addfieldpath="/administrator/components/com_content/models/fields">

I have it displayed and running when the slide is saved http://prntscr.com/can0vh , but when I create a new tab and click the button SELECT, which would select an article, it is not loaded in a modal window http://prntscr.com/can1gd

code that uses the example of this type
mod__product_.xml
<?xml version="1.0" encoding="utf-8"?>
<extension type="module" version="2.5.0" method="upgrade">
<config>
<fields name="params" addfieldpath="/modules/mod_virtuemart_product_tm/fields">
<fieldset name="basic">
<field
name="field_name"
type="subform"
formsource="modules/mod_virtuemart_product_tm/example.xml"
multiple="true"
min="0"
max="5"
label="Subform Field"
description="Subform Field Description"
layout="joomla.form.field.subform.repeatable" groupByFieldset="true"
/>
</fieldset>
</fields>
</config>
</extension>

and example.xml
<?xml version="1.0" encoding="UTF-8"?>
<form>
<fieldset name="section1" label="New Tabs for product modules">
<field
name="article_id"
type="modal_article"
required="true"
edit="true"
clear="false"
label="MOD_ARTICLES_SINGLE_FIELD_ART_ID"
/>
</fieldset>
</form>

For Me, the same occurs.
The results is that do not list any article, but loads the modal with a title:
"Select or change article"

@olejenya can you confirm Issue on latest Staging?

Set to "closed" on behalf of @franz-wohlkoenig by The JTracker Application at issues.joomla.org/joomla-cms/11805

This has been closed due to lack of response to the requests above – it can always be reopened in the future if it is updated.

This bug still exists in Joomla! 3.7.3

<fields name="params"> <fieldset name="basic" addfieldpath="/administrator/components/com_content/models/fields"> <field type="subform" name="articles" label="Nur Artikel" description="" message="" formsource="/modules/mod_articles_head/forms/articles.xml" multiple="true" min="0" max="50" layout="joomla.form.field.subform.repeatable-table" buttons="add,remove" groupByFieldset="false" /> ...

The subform XML:
<field name="item-article" type="modal_article" label="Artikel" class="input-xlarge" edit="true" clear="false" description="" /> ...

The result on the backend form: http://i.imgur.com/315ZWK0.png

And the empty modal: http://i.imgur.com/soa39lr.png

There is no Request sent (like: index.php?option=com_content&view=articles&layout=modal&tmpl=component&04a5525462f1005164e9a131ea1b59ed=1&function=jSelectArticle_jform_params_YOUR_FIELD_NAME) to the server when "Auswählen" in the subform is clicked.

I have the Exact same problem on Joomla 3.8.3
I tried @caru80 way and then I also tried to add the fieldset in the xml
<fieldset addfieldpath="/administrator/components/com_content/models/fields"> in the articles.xml file.

What I am actually trying to do is to let the user select multiple articles.
So in my custom module I have this field :

        <field name= "articles"
          description= "Select Articles"
          type= "subform"
          label= "Select Articles"
          required= "true"            
          formsource= "modules/mod_articleblocks/models/forms/articles.xml"
          multiple= "true"
          buttons= "add,remove,move"
          layout="joomla.form.field.subform.repeatable-table"
          groupByFieldset="false"/>

and the modules/mod_articleblocks/models/forms/articles.xml contains the following :

<?xml version="1.0" encoding="UTF-8"?>
<form>
  <fieldset name="Articles" addfieldpath="/administrator/components/com_content/models/fields">
    <field 
      name="article_id" 
      type="modal_article" 
      default="" 
      label="Select an article" 
      description="" />
  </fieldset>
</form>

The result is exactly the same as @caru80 says.

Same here

Same problem here, when a new row is add, try to click on select article, the request to server not occurs, and the modal appears blank

Hey guys. I don't know if you've looked in to the way the modal fields js works but it is fundamentally unsuited for use in repeatable subforms. I know how we can solve it but it's going to be basically a total rewrite of modal-fields.js among other things. Shall I?

Please do so for J4. Also do the field as a custom element with plain ES6 javascript

@dgrammatiko Well that actually sounds fun. Does Joomla 4 only support ES6-aware browsers or is there some transpiling happening somewhere?

Yes we write the code in an es6 file and do transpile it to es5. More info can be found here #20396.

#20788 is my attempt at fixing this. It's a pretty big change, maybe should discuss it and revise a bit before merging.

Closed as there is a PR