? ? ? ? Pending

User tests: Successful: Unsuccessful:

avatar richard67
richard67
28 Dec 2019

Pull Request for Issue #18736 .

Summary of Changes

This pull request (PR) adds back the special security check for "remote" database hosts at new installation which works in J3 and is described here: https://docs.joomla.org/Special:MyLanguage/J3.x:Secured_procedure_for_installing_Joomla_with_a_remote_database.

"remote" in quotes because this check also runs if a valid IP address or host name (short or FQDN) is given for a local database server (local means database server and web server = database client on the same host). The check assumes anything else than "localhost", "127.0.0.1" and "::1" to be a remote host. I think this was desired because as soon as forgotten Joomla installation is reachable from outside, it needs this security check.

Testing Instructions

Pre-condition: You either have a remote database server which can be accessed with a database client from your webserver, or you have both at the same host and set up your database sevrer so it listens to an IP address different to "127.0.0.1" (v4) or "::1" (v6), and optionally have a real host name different to "localhost" which can be resolved to such an IP address. If you have a remote database server you should also have a local one to test at the end that there is no security check it not necessary.

  1. Make sure your webserver has read and write access to the installation folder (should be the case for a clean git branch or an unzipped nightly build).

  2. Start a new installation of a 4.0-dev branch plus changes of this PR applied, i.e. either apply it with some git tool on a clean 4.0-dev branch, or apply it on an existing 4.0-dev installation e.g. with patchtester and then remove configuration.php and delete the database tables.

  3. Fill in all necessary data in the installation form and use something different to "localhost" and "127.0.0.1" and "::1" (see pre-conditions above).

  4. Click the "Install Joomla" button at the bottom.
    Result: See screenshot 1 in section "Expected result" below. The installation does not continue. An alert tells to delete a file created by Joomla.

  5. Delete the file as advised in the error alert.

  6. Click the "Install Joomla" button at the bottom.
    Result: See screenshot section "Actual result" below. The installation finish with success.

  7. Remove configuration.php, delete the database tables and delete the session cookie to clear session data.

  8. Make sure your webserver doesn't have write permission for the "installation" folder, e.g. by entering following in a Linux command shell as the user who is owner of the "installation" folder and with current directory = Joomla root: chmod -w installation. Or on Windows, add a permission which forbids anybody to write.

  9. Start again a new installation.

  10. Fill in all necessary data in the installation form and use something different to "localhost" and "127.0.0.1" and "::1" (see pre-conditions above).

  11. Click the "Install Joomla" button at the bottom.
    Result: See screenshot 2 in section "Expected result" below. The installation does not continue. An alert tells to upload or create a file with a given name.

  12. Change back permissions to like they were before step 8, e.g. on Linux: chmod +w installation.

  13. Upload or create the file as advised in the error alert.

  14. Click the "Install Joomla" button at the bottom.
    Result: See screenshot section "Actual result" below. The installation finish with success.

  15. Remove configuration.php, delete the database tables and delete the session cookie to clear session data.

  16. Start again a new installation.

  17. Fill in all necessary data in the installation form and this time use either "localhost" or "127.0.0.1" or "::1" (see pre-conditions above).

  18. Click the "Install Joomla" button at the bottom.
    Result: See screenshot section "Actual result" below. The installation finish with success without the security check for remote hosts.

Expected result

  1. If the file could be created by Joomla Installation:
    new-01

  2. If the file couldn't be created by Joomla Installation:
    new-02

  3. After all instructions shown in the alert have been completed: See actual result below.

Actual result

snap-02

Documentation Changes Required

None I think.

avatar richard67 richard67 - open - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 28 Dec 2019
Category Installation
avatar richard67 richard67 - change - 28 Dec 2019
Labels Added: ?
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67
richard67 - comment - 28 Dec 2019

@wilsonge Maybe the security check which is enabled and corrected by this PR should be moved from DatabaseModel.php, routine initialise, to file SetupModel.php, routine validateDbConnection? For installation using an existing database there will not be a difference as far as i could observe, but how is it with installation including creation of a new database? Have no idea how that should work, never worked for me.

avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar joomla-cms-bot joomla-cms-bot - change - 28 Dec 2019
Category Installation Installation Language & Strings
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar brianteeman
brianteeman - comment - 28 Dec 2019

Still don't understand why there are new strings

avatar richard67
richard67 - comment - 28 Dec 2019

Maybe check description and testing instructions then it is more clear?

avatar richard67 richard67 - change - 28 Dec 2019
Labels Added: ?
avatar joomla-cms-bot joomla-cms-bot - change - 28 Dec 2019
Category Installation Language & Strings Installation
avatar richard67
richard67 - comment - 28 Dec 2019

@brianteeman I've changed it back. If the user has to created the file manually, then the created file is enough check that the user has access. It does not need to make him delete it in this case. Will change testing instructions in a minute.

avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
Title
[4.0] [WiP] Add back remote database host security check at installation
[4.0] Add back remote database host security check at installation
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar richard67 richard67 - change - 28 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 28 Dec 2019
avatar brianteeman
brianteeman - comment - 29 Dec 2019
  1. Expected Result

image

That's not an error its an info message

  1. Could do with some text to say what to do after deleting the text file. Because of the changes in the installer in J4 its not as obvious anymore

  2. After deleting the file and clicking "Install Joomla" I get the spinning logo for a while and then I get
    image

BUT checking the remote database shows that the tables etc have been created

avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman

  1. Expected Result
    That's not an error its an info message

You mean both warning and error? Or only the error one, and the warning is ok?
From an error handling point of view, in my opinion everything which stops installation from proceeding should be an error. But you are right, from an UX point of view it might be confusing.

I suggest we collect more feedback and if people agree change it then. There is nothing easier than just changing the message type. This also can also be done later, so message type discussion should not stop this PR.

  1. Could do with some text to say what to do after deleting the text file. Because of the changes in the installer in J4 its not as obvious anymore

You mean to change the text in the alert of type error so they tell to click again the "Install Joomla" button after having deleted the file, and in case if the file had to be created manually click again the "Install Joomla" button after having created the file?

Please confirm if my understanding is right, or correct if not, and optionally propose the texts because for me as not native English speaker it is sometimes not easy to give enough information in as short as possible text.

When I'm sure that I've fully understood what you suggest, I'll implement the changes soon.

  1. After deleting the file and clicking "Install Joomla" I get the spinning logo for a while and then I get ...

Can you reproduce this and reproduce that it doesn't happen without this PR? I had this one time when testing, and it seemed to me that just the database was a bit slow at this time for some unknown reason.

So I'm not sure if it is related to my PR, but if it turns out that it is related, let me know and I'll propose an alternative (moving the checks from databae model to setup model) in another branch which could be tested for this issue.

avatar richard67
richard67 - comment - 29 Dec 2019

Meanwhile I'll click the "Update branch" button to see if drone fails again.

avatar richard67 richard67 - change - 29 Dec 2019
Labels Removed: ?
avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Maybe for your 2nd point:

To confirm that you are the owner of this website please leave this installation form open as it is and delete the file named \"%1$s\" we have created in the \"%2$s\" folder of your Joomla site. Then click the \"%3$s\" button again to continue with installation.

And for the other case:

We were not able to create the file. Please leave this installation form open as it is, manually create a file named \"%1$s\" and upload it to the \"%2$s\" folder of your Joomla site. Then click the \"%3$s\" button again to continue with installation.

("%3$s" will be the language text for the "Install Joomla" button, i.e. "Install Joomla" if English.

Is that ok, or is it too long, or is it bad English at some place?

avatar richard67
richard67 - comment - 29 Dec 2019

Hmm, done again failing ... that's more often than I got used to ... maybe there is really something wrong with this PR and some DB timeout, like @brianteeman has observed during his tests?

avatar richard67
richard67 - comment - 29 Dec 2019

@HLeithner Can it be that the mysql system tests fail because there is soething else than "localhost" or "127.0.0.1" or "::1" used as database host, and this magic environment variable which allows to bypass the security check is not set, so the test waits for someone to delete the file?

Or is "localhost" or "127.0.0.1" or "::1" used in our testing environment as database host? Then the security check will not be done and this is not the problem.

Do you know who knows that?

avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Regarding your 3rd point: If you can replicate that a few times with this PR, could you check if it is the same with PR #27371 ? It does the same as this PR here, just at another place in the code, functionality and testing is the same as here.

avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Regarding your points 1 and 2: Did you have something like this in mind?

new-proposal-01

new-proposal-02

My worry is that people put more attention on the warning than on the info, then they follow the link to the doc and read while their session for installation times out ;-)

Maybe we should bring both texts together to the info message, i.e. not show the warning but add its text to the bottom of the info message?

Please give feedback if you have time, it really is appreciated even if I don't always agree 100 %.

P.S.: Maybe in the second screen shot in the info box, the 1st sentence "We were not able to create the file." should be extended to "We were not able to create the file for website ownership verification." or "We were not able to create the file for the security check.", then it would be more clear. What do you think?

avatar brianteeman
brianteeman - comment - 29 Dec 2019

The remote installation failed again without this PR so maybe thts my connection and not related to this PR

avatar brianteeman
brianteeman - comment - 29 Dec 2019

It is not a notice it is information
keepalive is running on the installer so the session should not time out
simple rule about messages is the more text you write the less that people will read it.

avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Thanks for the info. I'll make a commit soon to avoid PHP warnings when the file can't be written, this will also speed the test up a bit in this case.

And then let's improve UI: Would be happy if you could check my above proposals and give feedback and maybe make better suggestions.

avatar richard67
richard67 - comment - 29 Dec 2019

It is not a notice it is information
keepalive is running on the installer so the session should not time out
simple rule about messages is the more text you write the less that people will read it.

@brianteeman Ok, thanks for feedback, will try it here with information. Regarding length of text: You mean my extensions to the texts so they explain what to be done as next are too long? I thought that this was that what you had in mind with your 2nd point, to explain the next step.

avatar richard67
richard67 - comment - 29 Dec 2019

It is not a notice it is information

@brianteeman I didn't find a predefined alert type "info" or "information", but I found "message", which is green. Was that what you had in mind?

new-proposal-03

avatar brianteeman
brianteeman - comment - 29 Dec 2019

<div class="alert alert-info">

avatar richard67
richard67 - comment - 29 Dec 2019

It is almost clear now that system tests are failing in drone because they use a host name which requires the security check to run, and the environment variable to bypass that is not set.

I more and more think that this exactly has been the reason why this security check has been switched off for J4 by commenting out the code.

avatar brianteeman
brianteeman - comment - 29 Dec 2019
INSTL_DATABASE_HOST_IS_NOT_LOCALHOST_CREATE_FILE="We were not able to create the file. Please manually create a file named \"%1$s\" and upload it to the \"%2$s\" folder of your Joomla site. Then select \"Install Joomla\" to continue. "
INSTL_DATABASE_HOST_IS_NOT_LOCALHOST_DELETE_FILE="To confirm that you are the owner of this website please delete the file named \"%1$s\" we have created in the \"%2$s\" folder of your Joomla site. Then select \"Install Joomla\" to continue. ""
INSTL_DATABASE_HOST_IS_NOT_LOCALHOST_GENERAL_MESSAGE="You are trying to use a database host which is not on your local server. For security reasons, you need to verify the ownership of your web hosting account. <a href=\"%s\">Please read the documentation</a> for more information."
avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Shouldn't I make the "Install Joomla" be a parameter and use the translated language text for that? The button text might be translated.

avatar brianteeman
brianteeman - comment - 29 Dec 2019

hopefully translators are not daft and they will realise but then again from experience they are so a param is better

avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman How can I get <div class="alert alert-info"> using Factory::getApplication()->enqueueMessage?

avatar joomla-cms-bot joomla-cms-bot - change - 29 Dec 2019
Category Installation Installation Language & Strings
avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman I see <div class="alert alert-info"> is something else:
div-alert-info

To use these would be a change on how IU works here. Here I use Factory::getApplication()->enqueueMessage to enqueue alert messages, like all other validations in the installation do.

What else do you suggest? Shall I change the type of the "error" alerts to "Notice" (blue)? Or to "Message" (green)? These are the options we have with enqueueMessage beside "warning" and "error".

avatar brianteeman
brianteeman - comment - 29 Dec 2019

notice or message will be fine

On Sun, 29 Dec 2019 at 14:42, Richard Fath notifications@github.com wrote:

@brianteeman https://github.com/brianteeman I see

is something else:
[image: div-alert-info]
https://user-images.githubusercontent.com/7413183/71558275-9c3a3780-2a51-11ea-94d7-7bcd717fa070.png

To use these would be a change on how IU works here. Here I use
Factory::getApplication()->enqueueMessage to enqueue alert messages, like
all other validations in the installation do.

What else do you suggest? Shall I change the type of the "error" alerts to
"Notice" (blue)? Or to "Message" (green)? These are the options we have
with enqueueMessage beside "warning" and "error".


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/joomla/joomla-cms/pull/27360?email_source=notifications&email_token=AAJ4P4OQZC5UJ6PBCFCDLHTQ3CZO3A5CNFSM4KAPHR2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHZA6YQ#issuecomment-569511778,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJ4P4O6IT627EFVUEUHEKDQ3CZO3ANCNFSM4KAPHR2A
.

--
Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
https://brian.teeman.net/ http://brian.teeman.net/

avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Then I will use notice (blue), because message (green) is only used for success messages.

On the other side, green gets more focus from me than blue, compared to the yellow of the warning, so maybe green would be better for me.

But then I think of people having red-green color blindness, so at the end blue is better I think.

avatar richard67 richard67 - change - 29 Dec 2019
Labels Added: ?
avatar richard67 richard67 - change - 29 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 29 Dec 2019
avatar richard67 richard67 - change - 29 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 29 Dec 2019
avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Is it ok now (see changed screenshots in description)? Or should we change the type of the warning to "notice", too?

I would also like to extend the text INSTL_DATABASE_HOST_IS_NOT_LOCALHOST_CREATE_FILE by the instruction to make the installation folder writable, or I have to revert my commit d1966dc , because it might confuse the user if he/she has created the file as requested but not changed folder permission so the check comes up with the same message again.

I know you don't like too long texts. But what is better? Extend it a bit so the user also corrects folder permissions, or change back the check in the code by reverting my commit so if the file is there installation goes on and succeeds, and only deleting the installation folder at the end might fail?

The last point I've decided to change it back to like it was, so language text is ok.

avatar richard67 richard67 - change - 29 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 29 Dec 2019
avatar richard67 richard67 - change - 29 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 29 Dec 2019
avatar joomla-cms-bot joomla-cms-bot - change - 29 Dec 2019
Category Installation Language & Strings Unit Tests Installation Language & Strings
avatar richard67 richard67 - change - 29 Dec 2019
The description was changed
avatar richard67 richard67 - edited - 29 Dec 2019
avatar richard67
richard67 - comment - 29 Dec 2019

Drone system tests fixed with last commit for .drone.yml.

@brianteeman All your suggestions are implemented. I'd be happy if you could find time to test. And thanks for your time with the reviews up to now, I really appreciate that.

avatar richard67
richard67 - comment - 29 Dec 2019

Drone passed ... is like bday and xmas and new year and easter at the same day ;-)

... ready for testing.

avatar brianteeman
brianteeman - comment - 29 Dec 2019

Seems all ok now but I cannot comment on the drone.yml changes

avatar richard67
richard67 - comment - 29 Dec 2019

@brianteeman Yes, the drone is something for Robert, he was pinged for review. Thanks for all, would be nice if you could give it a good test on the issue tracker.

avatar HLeithner HLeithner - change - 29 Dec 2019
Labels Added: ?
avatar HLeithner
HLeithner - comment - 29 Dec 2019

I signed your drone.yml

avatar richard67
richard67 - comment - 29 Dec 2019

@HLeithner Thanks, I've forgotten that.

avatar richard67
richard67 - comment - 30 Dec 2019

@zero-24 Please test if you can find some time.

avatar richard67
richard67 - comment - 30 Dec 2019

@wilsonge or @HLeithner Should I add the release blocker label to this PR because the issue #18736 solved by it has that label? And should it be added to the "[4.0] Installer" project? Label I can add myself, but project not.

avatar richard67
richard67 - comment - 30 Dec 2019

@Quy Thanks.

avatar alikon alikon - test_item - 31 Dec 2019 - Tested successfully
avatar alikon
alikon - comment - 31 Dec 2019

I have tested this item successfully on 6378972

i've simulated the "remote database host" via dcoker


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/27360.

avatar HLeithner HLeithner - test_item - 31 Dec 2019 - Tested successfully
avatar HLeithner
HLeithner - comment - 31 Dec 2019

I have tested this item successfully on 6378972

Works with r/w permission and with ro permissions.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/27360.

avatar richard67
richard67 - comment - 31 Dec 2019

Thanks a lot for reviews and testing. With 2 good tests it could be RTC, but there is a pending review request to @rdeutz as the code owner of the .drone.yml file, which is changed by this PR to define the environment variable necessary for skipping the security check in our testing environment for the system tests. Shall we wait for that review?

avatar alikon alikon - change - 1 Jan 2020
Status Pending Ready to Commit
avatar alikon
alikon - comment - 1 Jan 2020

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/27360.

avatar alikon
alikon - comment - 1 Jan 2020

setting RTC doesn't hurt, maybe a way to have a more quick review

avatar HLeithner
HLeithner - comment - 1 Jan 2020

The only other way I know to overcome the security check is add additional code to joomla and that makes no sense for testing. So I'm merging this thanks all

avatar HLeithner HLeithner - close - 1 Jan 2020
avatar HLeithner HLeithner - merge - 1 Jan 2020
avatar HLeithner HLeithner - change - 1 Jan 2020
Status Ready to Commit Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2020-01-01 10:44:53
Closed_By HLeithner
Labels Added: ?
avatar richard67
richard67 - comment - 1 Jan 2020

Thanks.

Add a Comment

Login with GitHub to post a comment