No Code Attached Yet
avatar andersonm66
andersonm66
11 Jun 2022

Steps to reproduce the issue

When installing Joomla 4 on a server running OpenLitespeed (in my case on a CyberPanel build on Ubuntu) and renaming htaccess.txt to .htaccess the administrator panel and some front end formatting no longer loads CSS code and is displayed unformatted. This is unusable ….

Expected result

Site should decompress CSS code correctly and format the backend correctly

Actual result

CSS code does not get decompressed and read and administrator panel is raw text with no formatting.

System information (as much as possible)

After some research and a lot of help from Nicholas at Akeeba Ltd we were able to discover that removing the second block of statements in the .htaccess file (The section with the GZip records) remedies the problem and allows you to run Joomla properly. Clearing caches in Joomla, on the server and in the browser may be required to reinstate correct formatting)

Additional comments

At present I have GZip compression disabled in the CyberPanel Litespeed settings and also in Joomla system config. I have removed the second block of htaccess code and Joomla 4 runs as it should. It appears that OpenLitespeed is incorrectly parsing the IF statements in the htaccess file.

I will update this post if I can get the caching running faster by enabling one or the other (in Joomla or on Litespeed) but the stated config with the edited htaccess file the site runs correctly.

It would be helpful if comments regarding this conflict were included above the second IF block in the htaccess file so that other users are aware of this conflict. In a perfect world Openlitespeed needs to modified to correctly parse the file but of course this is outside the scope of this GIT.

avatar andersonm66 andersonm66 - open - 11 Jun 2022
avatar joomla-cms-bot joomla-cms-bot - change - 11 Jun 2022
Labels Added: No Code Attached Yet
avatar joomla-cms-bot joomla-cms-bot - labeled - 11 Jun 2022
avatar nikosdion
nikosdion - comment - 11 Jun 2022

The real problem is that CyberPanel does not use LiteSpeed proper, it uses OpenLiteSpeed (the free of charge version of LiteSpeed).

OpenLiteSpeed does something really dumb. It only supports mod_rewrite rules in the .htaccess. By itself, that would be fine. However, what they also completely ignore any other directive including conditionals (<If>, <IfModule>, ...) and things like Header.

Now, remember that Joomla's htaccess.txt last block redirects .css and .js to .css.gz and .js.gz if these files exist and also sets the HTTP headers in a way that prevents the server from recompressing the already compressed files. This is all wrapped in IfModule conditionals so if you don't have both rewrite and header modules enabled it just won't fire.

Well, OpenLiteSpeed is supremely idiotic in that it ignores all the conditionals, executes the internal remapping of .css to .css.gz and .js to .js.gz, does not apply the Header directives and ends up trying to recompress the already compressed content, therefore breaking the site!

Since conditionals are not possible with OpenLiteSpeed there is nothing we can do in the htaccess.txt to prevent OpenLiteSpeed from breaking the site. At best, we can amend the comment block at the top to also include something along the lines of “If you are using OpenLiteSpeed (NOT LiteSpeed proper) as is the case e.g. with CyberPanel please remove this block as it will break your site due to the erroneous way OpenLiteSpeed parses .htaccess files”.

avatar brianteeman
brianteeman - comment - 11 Jun 2022

# READ THIS COMPLETELY IF YOU CHOOSE TO USE THIS FILE!

joomla-cms/htaccess.txt

Lines 130 to 133 in b882341

## If your site starts to look strange after enabling this, and you see
## ERR_CONTENT_DECODING_FAILED in your browser console network tab,
## then your server is already gzipping css and js files and you don't need this
## block enabled in your .htaccess

Is there really anything more that can be done?

avatar nikosdion
nikosdion - comment - 11 Jun 2022

@brianteeman Yes, there is. I inspected the results in the Network tab and it does not report ERR_DECODING_FAILED at least in Chromium-based browsers. It also doesn't help the user make the connection between OpenLiteSpeed and the problem. Right now the CyberPanel community thinks that Joomla 4 is broken and they talk about whether they need to switch to WordPress since Joomla doesn't seem to get fixed...

Mentioning the problematic software by name will most definitely help people understand that a. Joomla is aware of the problem, b. there is sod all we can do about it, and c. if you are using the affected software you really need to remove that part of the .htaccess because of a problem with your server software.

So, yes, I do think there's something more we can do about it than expecting the user to divine the meaning of technical jargon, especially when they are using "dummy mode" self-hosted server management software EXACTLY BECAUSE they do not understand technical jargon to begin with!

avatar brianteeman
brianteeman - comment - 11 Jun 2022

ok - it was a question. you answered. thank you. I dont have access to an openlitespeed server to check

avatar andersonm66
andersonm66 - comment - 11 Jun 2022

Speaking as a less educated user ... naming litespeed and/or cyberpanel specifically would have put me on the right track immediately. It took me some time to tie broken CSS to Gzip and then a lot of help from nicholas to actually work out the block had to be deleted....

avatar nikosdion
nikosdion - comment - 11 Jun 2022

Sorry, man, I spent my morning debugging this crazy thing.

Apparently the problem is also related to a bug fix LiteSpeed made 4 years ago as a response to a bug report of mine four years ago which introduced a change in functionality that plays a role in this bug appearing. Right now, regardless of whether you enable OpenLiteSpeed's GZip compression flag, it will forcibly enable GZip compression for all resources if your browser sends an Accept-Encoding HTTP header which contains gzip in it. This is of course wrong, if you disable the global compression flag it should do that so from the user's perspective OpenLiteSpeed's compression seems to have no effect to the problem which in turn misleads them into thinking Joomla is broken. Insane? Yeah. Luckily I was the one person who knew about this bug fix, how it was fixed (thanks to Rochen passing along the communication with the LiteSpeed folks four years ago) and that helped me understand what exactly is going on.

Regarding an environment, set up an Ubuntu Server 20.04 VM and run as root sh < (wget -O - http://cyberpanel.net/install.sh). When the installation is over access https://YOUR_VM_IP:8090 to get to the server control panel. Set up a new site and install Joomla 4. Rename htaccess.txt to .htaccess and you will see the problem.

avatar andersonm66
andersonm66 - comment - 11 Jun 2022

I also disabled GZip in the tuning options for Litespeed in the CyberPanel Web Interface.

avatar brianteeman
brianteeman - comment - 11 Jun 2022

@nikosdion no thanks - I would wather watch the F1 that build a new server. might try with scalahosting next week as they offer both litespeed and openlitespeed

avatar nikosdion
nikosdion - comment - 11 Jun 2022

@brianteeman No problem :) I was watching FP3 too!

Just a note, you will not reproduce the error on LiteSpeed proper — it understands the conditionals. I've been using LiteSpeed on my Rochen-hosted blog since Joomla 4.0.0-Beta2. You will only get this problem on OpenLiteSpeed which ignores the conditionals.

avatar brianteeman
brianteeman - comment - 17 Jun 2022

Thanks to scalahosting I now have an open litespeed server to test with.

avatar brianteeman
brianteeman - comment - 17 Jun 2022

So far all I can say is that Joomla does nto support openlitespeed. It kind of works with edge but it constantly gives errors with chrome

avatar nikosdion
nikosdion - comment - 17 Jun 2022

@brianteeman Remove the entire block for compressing the CSS and JS files. The site will magically work now.

If you look further up this discussion you will see I told you why:

OpenLiteSpeed does something really dumb. It only supports mod_rewrite rules in the .htaccess. By itself, that would be fine. However, what they also completely ignore any other directive including conditionals (, , ...) and things like Header.

It's not that Joomla doesn't support OpenLiteSpeed or that Joomla is doing anything wrong. The problem is that OpenLiteSpeed is doing something COMPLETELY WRONG, breaking the conditionals which work perfectly fine not only on Apache but also its own paid version, LiteSpeed. They are breaking sites on purpose to pressure people into paying for LiteSpeed. That's the other reason I think a name & shame is warranted. We don't want people to think that Joomla doesn't support OpenLiteSpeed when the root cause of the problem AND its solution have sod all to do with Joomla itself!

avatar brianteeman
brianteeman - comment - 17 Jun 2022

@brianteeman Remove the entire block for compressing the CSS and JS files. The site will magically work now.

already done that ;)

At first glance it is working correctly now in both browsers but I keep getting js errors in chrome that I do not get in edge.

avatar nikosdion
nikosdion - comment - 17 Jun 2022

@brianteeman If you have enabled LSCache you need to clear it. Also clear Chrome's cache; it may have cached the double-compressed files. It happened to me with Edge when I was testing.

avatar brianteeman
brianteeman - comment - 17 Jun 2022

been there and done that. but its hot and i need more coffee

However, what they also completely ignore any other directive including conditionals (, , ...)

that can't be exactly true because if you install their component for joomla it will add a block to the htaccess which contains a conditional

### LITESPEED_CACHE_START - Do not remove this line
<IfModule LiteSpeed>
CacheLookup on
## Uncomment the following directives if you has a separate mobile view
##RewriteEngine On
##RewriteCond %{HTTP_USER_AGENT} Mobile|Android|Silk/|Kindle|BlackBerry|Opera\ Mini|Opera\ Mobi [NC] 
##RewriteRule .* - [E=Cache-Control:vary=ismobile]
## Uncomment the following directives to enable login remember me
##RewriteCond %{HTTP_COOKIE} ^.*joomla_remember_me.*$
##RewriteCond %{HTTP_COOKIE} !^.*_lscache_vary.*$
##RewriteRule .* - [E=cache-control:no-cache]
</IfModule>
### LITESPEED_CACHE_END
avatar nikosdion
nikosdion - comment - 17 Jun 2022

Yeah, and what I am telling you is that the <IfModule> is completely ignored on the free of charge OpenLiteSpeed (BUT NOT the paid version, LiteSpeed proper). The entire thing works correctly on the paid LiteSpeed server. It does not work correctly on the free OpenLiteSpeed. These are two different products which work differently. This is a simple concept, why are you having so much trouble understanding it?!

I am not making this up, mate. It's in their official documentation: https://openlitespeed.org/kb/how-to-autoload-htaccess-with-openlitespeed/#Unsupported_Directives

OpenLiteSpeed currently only supports mod_rewrite rules from Apache

They really do mean it. NOTHING ELSE is supported. Instead of throwing an error — as they should — they just ignore it.

avatar nikosdion
nikosdion - comment - 17 Jun 2022

I mean, FFS, just try adding this at the top of your .htaccess

<IfModule mod_theHellThisEvenExists.c>
RewriteRule .* whattheactual.fsck
</IfModule>

Clearly, there is no such module as mod_theHellThisEvenExists.c loaded in OpenLiteSpeed. Yet, adding this conditional redirect makes your site inaccessible.

Try the same thing on LiteSpeed proper on Apache and your site does not break.

What more proof do you need that OpenLiteSpeed is broken?

avatar brianteeman
brianteeman - comment - 17 Jun 2022

I mean, FFS, just try adding this at the top of your .htaccess

Done

yet, adding this conditional redirect makes your site inaccessible.

See for yourself - the site is working as expected
https://testingthis.website

avatar richard67
richard67 - comment - 17 Jun 2022

So there is not much we can do if we don’t want to reduce our htaccess.txt to mod_rewrite rules only - what we for sure don’t want. But maybe we can provide some documentation on that and some reduced htaccess.txt for OpenLiteSpeed in that documentation? And maybe add in our htaccess.txt add a comment referring to OpenLiteSpeed and to that documentation?

avatar richard67
richard67 - comment - 17 Jun 2022

I mean, FFS, just try adding this at the top of your .htaccess

Done

yet, adding this conditional redirect makes your site inaccessible.

See for yourself - the site is working as expected https://testingthis.website

Hmm, maybe OpenLiteSpeed ignores the complete if block, I.e. also the stuff inside?

avatar andersonm66
andersonm66 - comment - 17 Jun 2022

I have just loaded chrome and tried on my website with no errors. If it is of any help I am happy for you to come and play on my server. I am building a new site so it’s not in production. I have not got any errors with my testing of chrome on ipad….

avatar brianteeman
brianteeman - comment - 17 Jun 2022

wow - this thing is really weird. One minute everything is working correctly on all browsers just with the defaults and the next minute it is borked.

will leave it for another day to dig further

avatar nikosdion
nikosdion - comment - 17 Jun 2022

@brianteeman That is the reason I did not reply to you immediately. I knew you never actually read the documentation page I linked you to.

Let's all RTFM together now (reminder: https://openlitespeed.org/kb/how-to-autoload-htaccess-with-openlitespeed/)

Let me quote the important bits.

As of v1.4.38, OpenLiteSpeed supports the ability to load .htaccess from directories and subdirectories automatically. The old ways of adding rewrite rules manually via the WebAdmin Console or vHost config will continue to work, but if you’re looking to manage your rewrite rules from .htaccess directly, we now have a solution for it, and it is the recommended way for rewrite rules going forward.
You can enable Auto Load from .htaccess from the server level, virtual host level or context level.

Your host has already done that. But! The last two paragraphs tell you just how incredibly counter-intuitive OLS is (emphasis mine):

Pleas (sic) only add rewrite rules supported by OLS to .htaccess. Unsupported directives will be ignored.
OpenLiteSpeed currently only supports mod_rewrite rules from Apache, so before making it read the .htaccess files, please convert
them to Apache’s mod_rewrite format. You can find some examples here. Remember to restart OLS, after making changes to your rewrite files, so that it can read the new ones.

Let me phrase this in slightly different words.

OLS does not actually support .htaccess files, it supports migrating rewrite rules it can understand it finds in eligible .htaccess files once upon server restart. Everything else in the .htaccess file is ignored. If you change your .htaccess files you need to restart the server for the migration to happen once again.

I'm pretty sure your host has a CRON job to auto-restart the server every several minutes. I know that GoDaddy are restarting the server every 30' or thereabouts. Your host probably does that every 15' to 20'. Remember how my backup restoration troubleshooting says that on some servers your restored site might break after a few minutes, if that's the case remove the .htaccess and wait for 30'? That's why, mate.

I'm not crazy, folks. I just happen to run into the weirdest server setups far more often than you do just because of the nature of the software I write and do support for. Moreover, I am stubborn enough to do hands-on troubleshooting to verify weird issues and doggedly pursue the root cause even if it takes years to divine which exact configuration makes it happen.

So, please, can we add a two line bloody comment in the bloody .htaccess to warn our users not to use the compressed static media on bloody OpenLiteSpeed / CyberPanel, referencing these two products by bloody name already?!!

avatar brianteeman
brianteeman - comment - 17 Jun 2022

there is nothing stopping you adding anything to the htaccess

avatar nikosdion
nikosdion - comment - 17 Jun 2022

But if I am the only one understanding the necessity how am I going to get two successful tests to merge it into the core product? I've had that problem plenty of times before, don't you think? ?

Thus, I first need to convince people I am not crazy, then help them understand what I understand, then make a PR. It's just that I keep bumping on problems caused by weird configurations and I am most definitely NOT a good teacher (at least I know where I suck at)...

avatar brianteeman
brianteeman - comment - 17 Jun 2022

yeah - if you had mentioned the cron stuff before it would have helped

avatar nikosdion
nikosdion - comment - 17 Jun 2022

Yup. That's why I am not a good teacher. I thought linking to the doc page which says it needs a restart would be enough. But because it's obvious to me and the weird way my brain works doesn't mean it's obvious to anybody else reading the same thing. Ugh...

avatar richard67
richard67 - comment - 17 Jun 2022

But if I am the only one understanding the necessity how am I going to get two successful tests to merge it into the core product?

@nikosdion I would support such a PR for such a comment in our htaccess.txt. If it is a problem to directly list the certain products there, we could link there do a docs page which does that.

avatar nikosdion
nikosdion - comment - 17 Jun 2022

OK, I'll do a PR a bit later, I'm about to go pick up my daughter coming back from school.

avatar brianteeman
brianteeman - comment - 17 Jun 2022

i just did one.

avatar richard67
richard67 - comment - 17 Jun 2022

Shall I close this issue?

avatar nikosdion
nikosdion - comment - 17 Jun 2022

@richard67 Yup, per standard practice you can close it since we have a PR.

avatar richard67 richard67 - close - 17 Jun 2022
avatar richard67
richard67 - comment - 17 Jun 2022

Closing as having a pull request. Please test #38084 . Thanks in advance.

avatar richard67 richard67 - change - 17 Jun 2022
Status New Closed
Closed_Date 0000-00-00 00:00:00 2022-06-17 15:20:47
Closed_By richard67
avatar richard67
richard67 - comment - 18 Jun 2022

P.S. and thanks to @nikosdion for your detailed analysis.

avatar andersonm66
andersonm66 - comment - 18 Jun 2022

Thanks to both of you for you assistance. Cheers

Add a Comment

Login with GitHub to post a comment