I uncovered this one troubleshooting an older component that had stopped working after a site was updated to the newest version of Joomla and PHP 7.
I was able to observe the issue because the code that had been written into the component utilized JModelList and it was pulling in the default list limit set in the Global Config of 20 for the site. The model's getListQuery() method was fine and the JDatabaseQuery object was being created along with a variable that had been bound to the query object (this was utilizing the new OCI8 Oracle Driver I had written over here in this PR: #12588).
Normally for the queries I typically write I'm not setting any sort of limit, but because of the use of JModelList in this case, the call to getItems() utilizes and sets a limit automatically.
Additionally, instead of using the select(), from(), where() from the query object, I was simply using $query->setQuery() and passing in the whole query directly, which was setting the internal sql property with the query string.
What I discovered was that this combination is problematic right now, at least for the Oracle driver (for both the proposed Oracle driver update in the PR above and the existing PDO version currently in the Joomla codebase since I based my version on what was already in the codebase last summer). You can't have a bound variable along with a limit/offset.
This is primarily due to the following line in the PDO driver:
https://github.com/joomla/joomla-cms/blob/staging/libraries/joomla/database/driver/pdo.php#L708
and the corresponding line in the new driver in the PR above:
https://github.com/orware/joomla-cms/blob/e1785ac2a96ef43892fb188ba981fc6198d0262f/libraries/joomla/database/driver/oracle.php#L731
The main reason why that line causes an issue is because processLimit() returns a string and in both of the lines above, that string is overwriting the query object so any variables that might been bound are then lost (executing the query results in an error).
On my end, I ended up fixing this by changing the line above to use the query's setLimit() method instead, which leaves the bound variables intact. However this doesn't fully fix the issue.
Since the raw query had been set directly within the query object, when the query is converted to a string it ignores the limit settings that have been set. This is because the __toString() method just returns the entire raw query:
https://github.com/joomla/joomla-cms/blob/staging/libraries/joomla/database/query.php#L223
With my notes further below about how Joomla 3.2.1 was still utilizing the setLimit() method correctly, I think this is why it never surfaced as an issue before, because even though a limit was being set via JModelList, it was then ignored when the query object was converted to a string and the query was executed so it ended up returning the expected full result set anyway.
However, since we want the correct behavior in all cases, if someone did set a raw query and did set a limit they should receive a limited result set as expected and that currently won't happen unless we also adjust the query object's __toString() method to include that information, or if we override that method with some additional code in at least the Oracle query classes (in order to prevent any potential issues for other drivers...this would have to be tested in a bit more detail).
The limited result set for the query with bound variables.
Query execution error (tested so far with the Oracle driver only, potentially also would apply to other drivers utilizing the JDatabaseDriverPdo class).
I'm testing this on PHP 7.0.x with Joomla 3.6.5 (though I'm pretty sure the same issue applies to 3.7).
This issue probably remained undetected for a while simply because for most of us that use these other drivers available in Joomla, if they are like me, we rarely need to set a limit directly in our code, and don't typically utilize the JModel functionality (we just write our own methods to retrieve the data we need from these other databases and integrate that into an extension we might be building for our organization).
What's a bit weird to me, is that this issue wasn't there before as far as I can tell, but mainly cropped up after I had upgraded the site to the latest Joomla version (I think it was on a 3.2.x release before I did the update) so even though that existing code was still using JModelList, it never seemed to be utilizing the Global Config list limit of 20 since it was always returning the full result as we expected in the past.
In taking a closer look at the Joomla 3.2.1 tagged release code, it appears that the issue must have been introduced at a later time, because the code then was correctly utilizing the query's setLimit() method:
https://github.com/joomla/joomla-cms/blob/3.2.1/libraries/joomla/database/driver/pdo.php#L697
Looks like the exact commit where this change was introduced was this one:
f11f4eb#diff-ec0f1d11cc854648938d5f4774f1fa0b
The gist @PhilETaylor has in that older commit is definitely related to the same issue I encountered above, but the fix that had been proposed in that older PR only works I think so long as no bound variables are being used. In the example Gist that Phil provided here:
https://gist.github.com/PhilETaylor/bdf4c5a29b6097098922
None of those examples show bound variables so the issue wouldn't have presented itself the way I ran into it above.
I'll try running some additional tests with my suggested fix above in the query object's __toString() method and Phil's gist to see what my results look like to compare hopefully later tonight.
Status | New | ⇒ | Needs Review |
Labels |
Added:
J3 Issue
|
after 2 years this can be closed @franz-wohlkoenig
Status | Needs Review | ⇒ | Closed |
Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2019-04-10 08:10:03 |
Closed_By | ⇒ | franz-wohlkoenig |
In regards to the second part of problem (making sure a raw query that has been set in the query object utilizes the limits that were set properly):
https://github.com/joomla/joomla-cms/blob/staging/libraries/joomla/database/query.php#L223
I modified the if statement in the line above to the following so that the limit values are correctly set (if no limits were set then it defaults to the current behavior of just returning the whole raw query:
The only thing I don't know is whether or not making this change inside of the main JDatabaseQuery object is the right place to do it, or if it should be done in one of the subclasses only instead.
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/14891.