is it related with #15466 ?

@AlexRed Yes, it is related but not exactly the same. That issue seems to be about the common situation that the file is deemed unsafe (maybe because an archive file legitimately contains files with forbidden extensions like .php, .java, or .py) and then the upload fails silently. That issue should definitely be addressed (personally, I like the idea of throwing exceptions in those cases and letting the calling function catch and deal with them).

What I'm mainly talking about here is a very rare but annoying problem where a zip file or other type of archive does not actually contain any files of forbidden types but some section of the binary data of the archive, when interpreted as text, randomly comes out as .py or .jar or one of the others. In that case, the upload also silently fails, of course, but it shouldn't fail at all. Even if it did fail with a message the user would end up confused because you'd see something like "This file couldn't be uploaded because it contains forbidden file types." but it, in fact, does not.

I guess what would be best is if Joomla actually peeked inside those kinds of files to check the contents rather than just reading over the archive.

Copying my comment from #8197
In addition to 3.6.2, I've been able to reproduce it on 3.7.2 with a zip-file containing one .mp4 file. The total file size was 188mb.

Just wanted to login and confirm this. I've tried uploading two zip files, one from a client (a SCORM package) and one of my own making containing some .mp4 files. Both are well over 100Mb and both fail when strstr() finds .py and .pl in them even though there are no such files in the archive. This is on Joomla 3.6.2. Any idea when we might get a patch? I've set fobidden_ext_in_content to false in order to have a functioning media manager.

@zaewin There is a workaround for this that will probably work. If you decompress the archive and then compress it again with a different app or with different settings, it will probably not randomly have a .py in it anywhere.

I'm not sure why the other issue got closed as it had been around longer... oh well.

@okonomiyaki3000 other Issue #8197?

@franz-wohlkoenig Precisely

@okonomiyaki3000 i didn't look which Issue is longer reported, i set a Link to duplicate Report.

In any case, it looks like this happens more often that I thought.

Since both:

• it is now implemented a good checking of file type (J3.7.2)
• the scaning binary files as being a text file is not meaningful

this scanning of content for binary files should be skipped, right ?

And besides PDF files, etc, for the case of binary files that are archives,

• any tools scanning such files would first decompress it in memory,

Joomla will scan it without decompressing / opening it ??

is this longer an Issue?

@franz-wohlkoenig Yes, tested on Joomla 3.7.5. I've tested it with a zip file at 252mb. Joomla fails silently to upload the file which is what I've been experiencing since 3.6.2.

There will always be the possibility of false positives and obviously the larger the file the more likely that is to happen. I really dont see anyway that this can be fixed so am closing it. It can always be reopened if I am incorrect or have misunderstood something..

@brianteeman I'm not sure if you've understood the fundamental problem here. The way this check is being done is completely wrong and makes no sense. These are binary files being opened as text and scanned for particular strings. The correct way is to open them with the appropriate function (maybe the PharData class works in most cases) and check their actual content. It should also be possible for a user to override this rule.

These are binary files being opened as text and scanned for particular strings

That is exactly correct - it is far too easy (and a common exploit) to disguise malicious code inside a binary file. Another popular cms has been shown to be vulnerable exactly because they didn't validate binary files

Could You please link to security advisory of this exploit?
In any case, this check should not disable harmless file uploads and fail silently. Ideas for improvements as stated above would be opt-out from this check and/or adding more context to binary file search, not just containing characters "php".

@brianteeman Sure, maybe they need to be scanned. But scanning them as text is not the way. There are ways to read these files properly.

Joomla is doing server-side mime-type check
and of course ignoring the client sent MIME Type

https://github.com/joomla/joomla-cms/blob/staging/libraries/src/Helper/MediaHelper.php#L62-L104

You cannot replace the full contents of the file, it will not pass (our current (linked above)) mime checks
(if this is false please someone with better knowledge, please provide a link, thanks !)

About MIME type checks, they are not file validator / not file parsers,
they only try to guess what the file type is
and they typically read a part of it (at least the first part of it) until they determine the file type
also changes in this happen between PHP versions, i do not know !

Now if you append a PHP or Javascript file to a PDF file then the returned mime check will still be
application/pdf

thus the file will continue to be regarded as PDF file
One of Joomla safety checks during reading of the file is if it contains <? so if you appended contents with the PHP start tag it will be blocked, otherwise it will pass with all the PHP code in there

is it exploitable to effect server ?
is it exploitable to effect client downloading the file ?
or some other similar case of detected mime like above that file is exploitable
(please someone with better knowledge, please provide a link, thanks !)

[EDIT]
i mean exploitable ... besides having access to server configuration (e.g. .htaccess) to declare the file extension as being a PHP executable ...

What are other CMS and frameworks doing more than the server-side MIME type checks ?
(please someone with knowledge of this, please provide a link to show the extra checks done, that are similar to what Joomla is doing, so that we learn their approach to this, thanks !)

Currently you can upload the content of .htaccess file if you name the file e.g. name some.txt
AddType application/x-httpd-php .jpg
AddType application/x-httpd-php .pdf

Do we need to care about this case too ?

Finally are these checks there also because the media manager can be configured to allow PHP , JS, etc files to be uploaded ? and we want to disallow such a thing ?

Currently you can upload the content of .htaccess file if you name the file e.g. name some.txt

AddType application/x-httpd-php .jpg
AddType application/x-httpd-php .pdf

We can't fix something like this if a user is so stupid

@brianteeman Sure, maybe they need to be scanned. But scanning them as text is not the way. There are ways to read these files properly.

Please share them then

I'm going to be a little too busy to do a PR for this for a little while but maybe someone else wants to start. Look at how you might use this stuff:

https://secure.php.net/manual/en/class.phardata.php
https://secure.php.net/manual/en/book.zip.php