okay, looks good but still not perfect

Personally I would prefer something "event based" without these 1000 callbacks (which still can be missed at some point of time).

Something like:

// Parent page
$(document).on('show.bs.modal', function(event) {
   Joomla.Event.listenOnce(event.target.iframeWindowContent, 'joomla:modal:doClose', function(){
      event.target.close();
   })
});

// Modal Iframe document
<button onclick="Joomla.Event.dispatch('joomla:modal:doClose')">close</button>

In 2 words: when user click close (or the script do request for close) it will dispatch custom event joomla:modal:doClose against it's window object (iframe window), this event will be catched by parent page, and in result the "parent code" will close the modal.

Profit here: you do not have to guess whether window.parent.Joomla.blabla even exists.
(btw, similar approach would be good as replacement for window.parent.JUserSelect(), jSelectArticle() and for other similar trap)

but

If we go the callback way Joomla.currentModal.get().close(), then we need small improve here.
as @dneukirchen already wrote in Glip.

It better to have Joomla.Modal object, which can be easily extended in future.

Joomla.Modal = {
  current: null,
  getCurrent: function() { return this.current; }
  setCurrent: function(element) { this.current = element; }
};

(btw, to support "multi modals" current can be transformed to array of opened modals and Joomla.Modal.getCurrent() will return only last item from array.)

Other notes, 2 potential issue here:

1. the code will fail if modal initialized somewhere externally, I mean without:

element.open = function () {
   return $self.modal('show');
};

2. modal element (not bootstrap) has own .open()/.close() methods, wich doing something else ?

the code will fail if modal initialized somewhere externally, I mean without element.open()

True, but we are creating an API so we can then say: use the API properly ?

modal element (not bootstrap) has own .open()/.close() methods, wich doing something else

Where did I saw that naming problem recently? Oh yeah in Javascript itself... Anyways the open/close can be named as jOpen/jClose (although I still like the open close, and letting people the don't want to use the Joomla's core UI components to figure out that themselves...

Now about the event driven approach: personally I don't mind but for newcomers it's harder to grasp the idea and the whole mechanics behind it. So depending what's our target audience (and I think that most people using/coding for joomla are not fluent in js) we should pick the best approach. If we can document this is=n plain english (that won't be me obviously) then we can go this way as well (and the dropping of the window.parent is a very tempting reason to follow this approach).

@dneukirchen so I updated this one to the specs we discussed in Koln.
@Fedik what do you think about this?

@dgrammatiko looks good, except 2 thing

1. you still have the code which use an old approach Joomla.currentModal.get().close()
2. you use Joomla.Modal.setCurrent(element), but then you access directly to the .current property Joomla.Modal.current.close(). This looks inconsistent.

you still have the code which use an old approach Joomla.currentModal.get().close()

Thanks, I missed those

but then you access directly to the .current property Joomla.Modal.current.close(). This looks inconsistent.

I thought that was what David wanted, I also like the idea of setters and getters as they are more explanatory, but I m also ok with this ?‍♂️

I think @dneukirchen meant to use Joomla.Modal instead of Joomla.currentModal,
but better ask him ?

With getters is better, because we can have more control over it,
example can trow an Error with explanation message, when user call getCurrent() before setCurrent()

I think @dneukirchen meant to use Joomla.Modal instead of Joomla.currentModal,
but better ask him ?

Yes, but thats only about naming.

Do you want me to use the getter instead of directly calling the stored value?

@Fedik @dneukirchen I made this codepen: https://codepen.io/dgrammatiko/pen/bMzpqB

So do we agree on correcting this from Joomla.Modal.current to Joomla.Modal.getCurrent() in combination to the exposed properties on the modals (onOpen and onClose)?

looks good,

do we agree on correcting this from Joomla.Modal.current to Joomla.Modal.getCurrent()

yes I think use Joomla.Modal.getCurrent() is better

Merged to continue the JS API.

Thank you @laoneo