? ? ? Pending

User tests: Successful: Unsuccessful:

avatar joomdonation
joomdonation
8 Mar 2021

Pull Request for Issue #30556.

Summary of Changes

This PR solves the Release Blocker issue #30556. It convert data of repeatable fields to the format accepted by subfield.

Testing Instructions

  1. Install Joomla 3.10.0. You can download the nightly build package here https://developer.joomla.org/nightlies/Joomla_3.10.0-alpha6-dev-Development-Full_Package.zip to install it.
  2. Create one or two repeatable field for articles.
  3. Create one or two articles, enter data for the created custom fields.
  4. Upgrade your site to latest Joomla 4.0-dev package (generate for this PR). Just download the updated package here https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/32611/downloads/41496/Joomla_4.0.0-beta8-dev+pr.32611-Development-Update_Package.zip , then go to Components -> Joomla ! Update, look Upload & Update tab and install that update package.
  5. Check the articles on upgraded site, edit these article, check and make sure data in Fields tab is there.

Actual result BEFORE applying this Pull Request

Data for repeatable fields not being migrated

Expected result AFTER applying this Pull Request

Data for repeatable fields are migrated properly

avatar joomdonation joomdonation - open - 8 Mar 2021
avatar joomdonation joomdonation - change - 8 Mar 2021
Status New Pending
avatar joomla-cms-bot joomla-cms-bot - change - 8 Mar 2021
Category Administration com_admin
avatar joomdonation joomdonation - change - 8 Mar 2021
Title
Migrate data for repeatable fields
[4.0] Migrate data for repeatable fields
avatar joomdonation joomdonation - edited - 8 Mar 2021
237dc74 8 Mar 2021 avatar joomdonation CS
avatar joomdonation joomdonation - change - 8 Mar 2021
Labels Added: ?
avatar joomdonation joomdonation - change - 8 Mar 2021
The description was changed
avatar joomdonation joomdonation - edited - 8 Mar 2021
avatar Quy
Quy - comment - 25 Mar 2021

@AndySDH Please consider testing. Thanks!

avatar obuisard obuisard - test_item - 26 Mar 2021 - Tested unsuccessfully
avatar obuisard
obuisard - comment - 26 Mar 2021

I have tested this item ? unsuccessfully on 237dc74

After update to Joomla 4, the Joomla 4 dashboard became unusable. The errors hint to the fact that the database was not updated in the process.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar joomdonation
joomdonation - comment - 26 Mar 2021

Hmm. Without this PR, does the update works well for you?

avatar obuisard
obuisard - comment - 26 Mar 2021

I have to try it out. Will keep you posted


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar joomdonation
joomdonation - comment - 26 Mar 2021

OK. Thanks @obuisard

avatar obuisard
obuisard - comment - 26 Mar 2021

I can confirm: the error does not come from the PR, the update from 3.10 alpha 6 dev to Joomla 4 beta 8 dev (update package) fails.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar joomdonation
joomdonation - comment - 26 Mar 2021

@obuisard Maybe you can raise a new issue for that? If you can let us know step by step what you did and got that error, that would help us to check to see why it happens and get it sorted

avatar obuisard
obuisard - comment - 26 Mar 2021

I followed the exact test instructions.
Except that upon update, the files are updated, not the database.
In the console, I get 1093 You can't specify target table 'y2jza_template_styles' for update in FROM clause
When I look at the database, the template_styles table has not been updated.
I will test with 'official' releases, not the nightlies to see if that is inherent to the fact that they are nightlies (and may be unstable).
If updating from 3.10 alpha 5 to Joomla 4 beta 7 fails, I will open a new bug issue.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar sksuryan
sksuryan - comment - 26 Mar 2021

I faced the same issue that @obuisard mentioned with the 4.0-dev nightly package without the PR as well as with the PR. Although, the article section in the one without the PR seems to be usable now, I don't know how that happened. Sadly, I can't say the same about the one with PR because I deleted that installation.

Since then, I have created two new Joomla 3 fresh installations and upgraded them with the 4.0-dev nightly with PR package and tested both of them successfully(one on Microsoft Edge, other on Firefox Private Window).

avatar sksuryan sksuryan - test_item - 26 Mar 2021 - Tested successfully
avatar sksuryan
sksuryan - comment - 26 Mar 2021

I have tested this item successfully on 237dc74


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar RickR2H RickR2H - test_item - 27 Mar 2021 - Tested unsuccessfully
avatar RickR2H
RickR2H - comment - 27 Mar 2021

I have tested this item ? unsuccessfully on 237dc74

Using the provided files the update works but, the Joomla article layout is missing all settings tabs. This was before and after the patch after update to J4! So in both situations I couldn't check if the fields are there or not. I can confirm that the custom fields values are present in the DB after upgrade.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar joomdonation
joomdonation - comment - 27 Mar 2021

@RickR2H This PR just do the data migration, so the issue with Joomla articles layout might be un-related. After the update, could you please update your site to latest Joomla 4 nightly build package which can be downloaded here https://developer.joomla.org/nightly-builds.html , then check it again to see if articles layout is sorted?

avatar obuisard
obuisard - comment - 27 Mar 2021

No matter what I have tried, I was unable to successfully update from 3.9.10 Alpha 5 or 6-dev to 4.0 beta8-dev or 4.0 beta8-dev with the patch. I am going to file a bug report or is Joomla 3.10 not even ready for a direct update to Joomla 4?


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar richard67
richard67 - comment - 27 Mar 2021

No matter what I have tried, I was unable to successfully update from 3.9.10 Alpha 5 or 6-dev to 4.0 beta8-dev or 4.0 beta8-dev with the patch. I am going to file a bug report or is Joomla 3.10 not even ready for a direct update to Joomla 4?

@obuisard Updating 3.9.10 Alpha 5 to 4.0 beta8-dev should work. If not, open an issue please. Thanks in advance.

avatar joomdonation joomdonation - change - 2 Apr 2021
Labels Added: ?
avatar joomdonation joomdonation - change - 2 Apr 2021
The description was changed
avatar joomdonation joomdonation - edited - 2 Apr 2021
avatar joomdonation
joomdonation - comment - 2 Apr 2021

Today, I synchronize this branch with latest 4.0-dev. Tested it again and it worked. Update went well, data is migrated properly. Could you please help testing again one more time?

avatar joomdonation joomdonation - change - 2 Apr 2021
Labels Added: ?
Removed: ?
avatar joomdonation joomdonation - change - 2 Apr 2021
Labels Added: ?
Removed: ?
avatar alikon
alikon - comment - 2 Apr 2021

.. update works but, the article layout is missing all settings tabs. (only in the artcile that have had repeteable before migration)

avatar joomdonation
joomdonation - comment - 2 Apr 2021

Could you show me a screenshot? Strange it is working well for me. Here is screenshot for my test article
article

avatar alikon
alikon - comment - 2 Apr 2021

sure
this is when edit the article that have fields before migration no tabs
Screenshot from 2021-04-02 11-02-16

this is when edit the article with no fields before migration tabs

Screenshot from 2021-04-02 11-04-57

avatar joomdonation
joomdonation - comment - 2 Apr 2021

I have no idea. Maybe I should setup test articles with and without fields and testing again. Or maybe you can send export and send database of the updated site to me for debugging purpose? I sent you a message via Glip

avatar alikon
alikon - comment - 2 Apr 2021

weird
Screenshot from 2021-04-02 11-26-29

avatar joomdonation
joomdonation - comment - 2 Apr 2021

Strange, too. I use your database and here is the article screenshot, no error like in your test. Maybe the difference is I'm using Window ?

How did you perform updating? I download the package here https://ci.joomla.org/artifacts/joomla/joomla-cms/4.0-dev/32611/downloads/41493/Joomla_4.0.0-beta8-dev+pr.32611-Development-Update_Package.zip, then go to Components -> Joomla Update, Upload & Update tab to upload the package and update

article

avatar astridx
astridx - comment - 2 Apr 2021

I have the same error like @alikon on Ubuntu. systeminfo-2021-04-02T13 54 44+02 00.txt

Articles Fields test Administration

Articles New test Administration

Joomla Update test Administration

Articles Edit test Administration

avatar joomdonation
joomdonation - comment - 2 Apr 2021

@astridx I'm unsure what's wrong with the update as when I use the database from @alikon , import it to my site, the article is working well. Maybe you can confirm that by configure your fresh Joomla 4 installation to use the migrated database?

I mean the data migration process works OK, I just don't know what part of the update process causing that error (the js error from browser console looks strange to me)

avatar richard67
richard67 - comment - 2 Apr 2021

Our updates currently are missing the deletion of this or that file or folder on update, i.e. after an update there might be files still present which should be removed. It's the usual thing with the files and folders deletion in com_admin's script.php not being up to date. It's a work in progress to improve that maintenance task and automate it as much as possible. It's just my guess that something like that might cause the issues.

avatar astridx
astridx - comment - 2 Apr 2021

@astridx I'm unsure what's wrong with the update as when I use the database from @alikon , import it to my site, the article is working well. Maybe you can confirm that by configure your fresh Joomla 4 installation to use the migrated database?

I mean the data migration process works OK, I just don't know what part of the update process causing that error (the js error from browser console looks strange to me)

I can confirm this. I just cloned a new 4.0-dev branch and copied the configuration.php from the failed test.
Now I can edit all tabs of the article.

avatar joomdonation
joomdonation - comment - 2 Apr 2021

If so, please check the migrated data. If the migrated data for repeatable form fields is OK, I would say that this is a successful test.

The error which you are having here could cause by some files left during update (as explained by @richard67) and it is unrelated to this PR. This PR just focus on data migration only

avatar astridx
astridx - comment - 2 Apr 2021

The data was correct, there were only two text fields with simple texts.

avatar joomdonation
joomdonation - comment - 2 Apr 2021

If so, I would say that it is a successful test?

avatar alikon
alikon - comment - 2 Apr 2021

yes for me

avatar alikon alikon - test_item - 2 Apr 2021 - Tested successfully
avatar alikon
alikon - comment - 2 Apr 2021

I have tested this item successfully on 5af3c9a


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar richard67
richard67 - comment - 2 Apr 2021

@astridx Would you say your test was successful? If yes, please mark your test result. Thanks in advance.

avatar HLeithner HLeithner - change - 9 Apr 2021
The description was changed
avatar HLeithner HLeithner - edited - 9 Apr 2021
avatar richard67 richard67 - alter_testresult - 9 Apr 2021 - sksuryan: Tested successfully
avatar richard67 richard67 - change - 9 Apr 2021
Status Pending Ready to Commit
avatar richard67
richard67 - comment - 9 Apr 2021

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/32611.

avatar richard67
richard67 - comment - 9 Apr 2021

I've restored @sksuryan 's successful test in the tracker so it's properly counted. The changes after that test which have invalidated the test result was only code style and quote names added.

avatar wilsonge wilsonge - change - 9 Apr 2021
Status Ready to Commit Fixed in Code Base
Closed_Date 0000-00-00 00:00:00 2021-04-09 18:06:08
Closed_By wilsonge
Labels Added: ? ?
Removed: ?
avatar wilsonge wilsonge - close - 9 Apr 2021
avatar wilsonge wilsonge - merge - 9 Apr 2021
avatar wilsonge
wilsonge - comment - 9 Apr 2021

OK let's get this in. Thanks!

avatar HLeithner
HLeithner - comment - 9 Apr 2021

I tested this and it looks ok only error message I get which I'm not sure if it's related after update for a media file:

<b>Warning</b>: Illegal string offset 'imagefile' in <b>/plugins/fields/media/tmpl/media.php</b> on line <b>31</b><br />
 

avatar joomdonation
joomdonation - comment - 10 Apr 2021

@HLeithner : Does the media file a part of repeatble field which we are migrating data to SubField?

avatar HLeithner
HLeithner - comment - 10 Apr 2021
avatar joomdonation
joomdonation - comment - 10 Apr 2021

@HLeithner Could you send me a copy of the migrated database (maybe via Glip) so that I can debug (prefer sql format)?

avatar HLeithner
HLeithner - comment - 10 Apr 2021

sure

avatar joomdonation
joomdonation - comment - 10 Apr 2021

@HLeithner I can confirm the issue with media field type. The reason is because in Joomla 3, data for the field is stored in string format (something like images\powered_by.png) but in Joomla 4, we store it in JSON format (something like {"imagefile":"images/powered_by.png","alt_text":""}

Look like when media field is used alone, the plugin do that conversion itself https://github.com/joomla/joomla-cms/blob/4.0-dev/plugins/fields/media/media.php#L78-L88 . But when the field is used in subfields, it is not converted yet

Not sure if we can do that conversion somehow when media is used on subfields? Do you have any idea ? Or maybe @laoneo know a way?

Of course we can do that format conversion in the migration code but I'm unsure if that's the right away?

One more thing, we might have problem with media field (uses alone, not related to this PR). In Joomla 3, we just store path to the image but in Joomla 4, it also store size of the image {"imagefile":"images/joomla_black.png?joomla_image_width=225&joomla_image_height=50","alt_text":""} . From my quick test, without joomla_image_width and joomla_image_height, the image is not being displayed (with and height of the img tag set to 0 for some reasons)

avatar HLeithner
HLeithner - comment - 10 Apr 2021

I think that was a @dgrammatiko thing for lazyloading, maybe he can help us here.

avatar dgrammatiko
dgrammatiko - comment - 10 Apr 2021

I think that was a @dgrammatiko thing for lazyloading, maybe he can help us here.

As I read the comment I think this has nothing to do with lazyloading but rather the fact that the particular custom field now stores also the alt data. The lazyloading is still done with a string (appending some query params, which was hugely controvesial and there were arguments) but we kept it as a string for the shake of B/C. I guess Brian could have some insights on the implementation of the alt attribute (I don't think I was involved in that)

avatar dgrammatiko
dgrammatiko - comment - 10 Apr 2021

From my quick test, without joomla_image_width and joomla_image_height, the image is not being displayed (with and height of the img tag set to 0 for some reasons)

The image needs to be processed with

public static function cleanImageURL($url)
and if there are width and hight set the lazyloading tag, etc. I thought I already did that
$img = HTMLHelper::cleanImageURL($value['imagefile']);

Edit the implementation is buggy, there needs to be a check for width and height before setting the attributes otherwise the defaults (for images without extra params) are 0, will do a PR to fix this

@joomdonation can you check #33093

avatar joomdonation
joomdonation - comment - 11 Apr 2021

@HLeithner The warning should be solved by PR #33094 , could you please take a look at it?

Add a Comment

Login with GitHub to post a comment