we shouldn't try and modify articles ourselves

We should never touch a users data

Can we close this one in favor of #20153?

although I agree that html5 is the better way because of the difficulties with upgrades then yes it should be closed. My 2c only

First we need to look at what will happen if we remove the caption.js:
All articles created since Joomla 3.2.2 (released 2104-02-06) already use the figure element when the caption was created using the editor button. For those there will be no change in behavior.
Articles created before that time (over 4 years old!) will loose the caption. The text will instead still be in the regular title attribute and the browser will show it when hovering over the picture. So it's not lost, but it appears different.

Imho, that is a minor thing which doesn't need solving. But if it is important for a site that the captions appear, they can still include the existing caption.js into their template or use a plugin which does something similar.

From my pov this can be closed.

Ok. Going to close this one and merge the caption.js removal. I'v also added the documentation required label on the pr that we do not forget to document that break.