[#22752] - Action logs CSV export with Generator support
- Fixed in Code Base
- 24 Oct 2018
- Medium
- Build: staging
- # 22752
- Diff
- mbabker:csv-with-generator-because-old-php-sucks
User tests: Successful: Unsuccessful:
Pull Request for Issue #22700
Summary of Changes
This builds on #22750 to add a separate code path for PHP 5.5+ users to use the more memory efficient Generators for the CSV data. As we are building a data set that is iterated over one time only, this fits the use case for a Generator, unfortunately it is a feature newer than our minimum requirements.
Generally, we would not accept a solution that creates different code paths for different PHP versions, but I think we're reaching a point where it's becoming OK to create special exemptions for special circumstances, this being one of them.
- Of sites reporting stats and running 3.8.13, 2.2% of those sites are using PHP 5.3 or 5.4 (and 5.2% of all sites having sent an update in the last 90 days across all versions), therefore this code path is for the benefit of an overwhelming majority of users
- While this creates a differing return type depending on the PHP version, the intended use case for it (doing a one-time
foreachloop over the result) is not impacted
See #22750 for info on the first commit of this PR.
Testing Instructions
Create a large database table of action logs to export, try to export across PHP versions. It should work for all versions and on 5.5 and newer use the memory optimized code path. Using the database @PhilETaylor supplied in #22700 I cannot get the export to complete on a PHP 7.2 system with 256M memory limit, using a Generator I can.
| Status | New | ⇒ | Pending |
joomla-cms-bot
- | Category | ⇒ | Repository Administration Language & Strings External Library Composer Change Libraries Front End Plugins |
Sounds like a good plan to me need to be tested.
What are the stats on MysqlPDO users verses other database layers? There seems to be a PHP 5.3 compatible way of doing large CSV exports based on PDO Statements... Maybe we could provide large exports for mysql and mysqlpdo down to 5.3 like this?
What about the other database types? Postgres etc...
Maybe we could remove the "web download" of "all rows" and only allow selecting of "some rows"?
Maybe move the "export all rows" to the CLI app ?
What are the stats on MysqlPDO users verses other database layers?
Of the three MySQL drivers it has the lowest use. In order; ext/mysqli, ext/mysql, ext/pdo. PDO MySQL use is still higher than all non-MySQL drivers combined though.
Maybe move the "export all rows" to the CLI app ?
Probably a good idea to provide a CLI script for this and integrate it "properly" into the Console application when merged up to 4.0.
Of the three MySQL drivers it has the lowest use
So we can rule that out then...
Probably a good idea to provide a CLI script for this and integrate it "properly" into the Console application when merged up to 4.0.
My thinking is that it's going to be impossible to provide an "on click download" of 1,000,000 rows of an action log (and YES it will grow over time to be that big on big sites incorrectly configured and badly managed) with a consistent user experience, a consistent "non-hacky" code true to Joomla core, and a single click, and using http.
Especially on the myriad of "different" (read: bad) webhost configurations out there.
This is also a report that probably is only needed occasionally, and could easily be the job of the person with command line access (or a person who could probably have used phpMyAdmin (or other) tool for the job anyway...)
Or... why not rip out the whole "export of all rows" feature completely?
Joomla doesn't provide exports of unlimited rows of anything else IIRC. So why start now?
Or... why not rip out the whole "export of all rows" feature completely?
Joomla doesn't provide exports of unlimited rows of anything else IIRC. So why start now?
In this case, the action logging system acts as an audit log of sorts and it makes sense to have a full export capability here. No other core system has a "need" for a bulk export capability like this one. Unfortunately, it's just not easy to get a performant system in place to serve the web facing admin app to pull it off.
| Labels |
Added:
?
?
?
|
||
| Status | Pending | ⇒ | Fixed in Code Base |
| Closed_Date | 0000-00-00 00:00:00 | ⇒ | 2018-10-24 22:00:11 |
| Closed_By | ⇒ | mbabker |
@joomla/cms-maintainers feedback is needed here...