The Bootstrap 5 Modal documentation has sections Optional sizes and Fullscreen modal. These sections describe CSS classes that are added to <div class="modal-dialog">. Using these classes, you can change the size of the modal window, taking into account responsive breakpoints.
So, using the .modal-sm, .modal-lg or .modal-xl class, you can set the size of the modal window. Using the .modal-fullscreen classset, you can set the full-screen size of the modal window.

In Joomla 4, the default layout of a modal window has the .modal-lg class, and the window dimensions are described by the optional parameters bodyHeight and modalWidth, each of which is a number. Units of measurement - viewport units. For example, if 'modalWidth' => 70 is specified in the array of parameters when calling LayoutHelper, then the CSS class jviewport-width70 will be added to the <div class="modal-dialog modal-lg">.
The set of jviewport-width* classes is defined in the CSS files of the Atum and Cassiopeia template.

Situation: the frontend template uses Bootstrap 5, the module wants to use a ready-made modal layout and you need to specify the .modal-fullscreen class for it. However, this is not possible with the current layout implementation. Therefore, you either need to copy a set of jviewport-width* classes into your template, or copy the HTML code of the modal window into the module and specify all the necessary CSS classes in it. But why do this if there is a ready-made modal window layout in the Joomla 4 core?

Summary of Changes

Change file layouts/libraries/html/bootstrap/modal/main.php.
An optional modalCss (string) parameter has been added to the params array. In this parameter, you can specify the necessary CSS classes. If the parameter is omitted, then by default <div class="modal-dialog"> has the .modal-lg class, just as it was before. If the modalWidth and modalCss parameters are specified at the same time, then the modalCss parameter has a higher priority. <div class="modal-dialog"> will be rendered with the modalCss parameter. modalWidth will be ignored. Thus, nothing will change for the Joomla core, since this parameter is not used, backward compatibility is not broken.

Testing Instructions

1. Try to render a modal window by default Joomla core way and see then nothing is changed:

<?php 
    $ModalData = [
        'selector' => 'AnySelectorModal',
        'params'   => [
            'title'      => 'Modal header text',
            'footer'     => 'Modal footer text',
            'height'     => '400px', // height of the <iframe> containing the remote resource
            'width'      => '800px', // width of the <iframe> containing the remote resource
            'bodyHeight' => 70, // Optional height of the modal body in viewport units (vh)
            'modalWidth' => 80, // Optional width of the modal in viewport units (vh)
        ],
        'body' => 'Modal body content'
    ];
   
    echo LayoutHelper::render('libraries.html.bootstrap.modal.main', $ModalData); 
    ?>

We got output HTML with <div class="modal-body jviewport-height70">.
2. Then try to use modalCss instead modalWidth or modalCss only:

<?php 
    $ModalData = [
        'selector' => 'AnySelectorModal',
        'params'   => [
            'title'      => 'Modal header text',
            'footer'     => 'Modal footer text',
            'height'     => '400px', // height of the <iframe> containing the remote resource
            'width'      => '800px', // width of the <iframe> containing the remote resource
            'bodyHeight' => 70, // Optional height of the modal body in viewport units (vh)
            'modalCss' => 'modal-fullscreen',
        ],
        'body' => 'Modal body content'
    ];
   
    echo LayoutHelper::render('libraries.html.bootstrap.modal.main', $ModalData); 
    ?>

And find <div class="modal-dialog modal-fullscreen"> has rendered.
3. Try to use both parameters together.

<?php 
    $ModalData = [
        'selector' => 'AnySelectorModal',
        'params'   => [
            'title'      => 'Modal header text',
            'footer'     => 'Modal footer text',
            'height'     => '400px', // height of the <iframe> containing the remote resource
            'width'      => '800px', // width of the <iframe> containing the remote resource
            'bodyHeight' => 70, // Optional height of the modal body in viewport units (vh)
            'modalWidth' => 80, // Optional width of the modal in viewport units (vh)
            'modalCss' => 'modal-fullscreen',
        ],
        'body' => 'Modal body content'
    ];
   
    echo LayoutHelper::render('libraries.html.bootstrap.modal.main', $ModalData); 
    ?>

We got a <div class="modal-dialog modal-fullscreen"> . modalWidth parameter was ignored.

Actual result BEFORE applying this Pull Request

We cannot specify Bootstrap CSS classes for optional sizes and fullscreen mode to modal window layout.

Expected result AFTER applying this Pull Request

Now we can do it and backward compatibility is not broken.