No Code Attached Yet
avatar stutteringp0et
stutteringp0et
14 Dec 2017

SMTP delays are painfully obvious to end users. They click "Send" on a contact form and they wait. They click "Register" and they wait. They click "Reset Password" and they wait. Sometimes it's a few seconds, sometimes longer.

If you've got a component that sends email, maybe you notice it more. I have one that sends several emails out every time particular events occur. Even sending via SMTP to a server I control - the delay is noticeable.

I've seen this done in other systems, and I think it might be a good idea to implement something like this in Joomla - as an option.

An email queue is a database table that holds outgoing messages so they may be sent at regular intervals via CRON job. I would say that the entire queue is sent at each CRON interval, but it would probably be smart to implement an option to limit the number of messages sent in each run. If a user initiates a mail event - that email is added to the queue and sent out when it reaches the head of the queue.

I think the benefits should be obvious.

From a user perspective, this would make the site appear faster.
Possibility for prioritization (new user email is higher priority than lost password email, and all system emails take priority over 3rd party extension emails).
Temporary errors could initiate a retry delay.

The only drawback I can think of would be that there would be a slight delay in mail delivery. Using a 1 minute interval, that delay should normally be less than 1 minute.

If implemented as a delivery option, users who don't have access to or don't understand CRON would experience things as they are. Any administrator capable of enabling this feature could deliver a better user experience.

What are your thoughts?

Votes

# of Users Experiencing Issue
1/1
Average Importance Score
5.00

avatar stutteringp0et stutteringp0et - open - 14 Dec 2017
avatar joomla-cms-bot joomla-cms-bot - change - 14 Dec 2017
Labels Added: No Code Attached Yet
avatar joomla-cms-bot joomla-cms-bot - labeled - 14 Dec 2017
avatar stutteringp0et stutteringp0et - change - 14 Dec 2017
The description was changed
avatar stutteringp0et stutteringp0et - edited - 14 Dec 2017
avatar stutteringp0et stutteringp0et - change - 14 Dec 2017
The description was changed
avatar stutteringp0et stutteringp0et - edited - 14 Dec 2017
avatar brianteeman
brianteeman - comment - 14 Dec 2017

wouldnt a server that operates smtp mail throttling also place restrictions on how often you can run a cron?

avatar mbabker
mbabker - comment - 14 Dec 2017

Probably. But there're are more reasons to support a job queue than just response times. In the case of mail sending, failed jobs can be retried and presumably we could more easily wire up logging tools in the CLI environment than over web.

Things like this are exactly why the CLI console package is integrated into 4.0. It exposes the possibility for core and extension builders to build better tooling.

avatar stutteringp0et
stutteringp0et - comment - 14 Dec 2017

@brianteeman That's one reason it should be an optional configuration - for servers/hosts that aren't capable.

Other reasons include, admins who don't know how to initiate a CRON job.

For systems that throttle SMTP, this should be able to accommodate. As each message is sent, it's removed from the queue. If the cron hits a throttle, the mail isn't sent and the message can remain in queue until the next run.

@mbabker - that was one of my reasons above - temporary errors could initiate a retry delay.

I use CLI for all sorts of stuff. It's great to have that capability.

avatar stutteringp0et
stutteringp0et - comment - 14 Dec 2017

@mbabker - I'm glad you mentioned logging - because this could be used to log and report on outgoing messages and smtp performance.

avatar stutteringp0et
stutteringp0et - comment - 15 Dec 2017

With some quick research, I found that PHPMailer has a facility to output a message unsent as a complete RFC822 message.

$mailer->preSend();
$message = $mailer->getSentMIMEMessage();
avatar stutteringp0et
stutteringp0et - comment - 15 Dec 2017

Looks like the mailSend()/smtpSend()/sendmailSend() methods only require a $header and $body - which is easily obtainable via $mailer->createHeader() and $mailer->createBody() without the need to run $mailer->preSend(). So 2 database fields and we have everything we need to send a message later.

avatar stutteringp0et
stutteringp0et - comment - 15 Dec 2017

Some testing reveals that intercepting a message in Mail::Send() and storing it in the database is pretty easy to do.

avatar franz-wohlkoenig franz-wohlkoenig - change - 15 Dec 2017
Category com_plugins UI/UX
avatar franz-wohlkoenig franz-wohlkoenig - change - 15 Dec 2017
Status New Discussion
avatar tonypartridge
tonypartridge - comment - 18 Dec 2017

I know a guy who’s written a mailbank type component/plugin and planned on releasing it but never did.

It was also something I wanted to add to the Joomla! Core. Allowing a log of all emails sent/not sent would be very useful and having a single cron/mailer script but also be useful, I.e. addtoque method instead of just send.

avatar stutteringp0et
stutteringp0et - comment - 18 Dec 2017

My thoughts would be to modify JMail to queue if configured, otherwise send normally. That way it's transparent to extension developers - they send normally and the system handles the message (queue or send)

avatar tonypartridge
tonypartridge - comment - 18 Dec 2017

Sounds great to me and exactly what I would do.

On 18 Dec 2017, 07:33 +0000, Michael Richey notifications@github.com, wrote:

My thoughts would be to modify JMail to queue if configured, otherwise send normally. That way it's transparent to extension developers - they send normally and the system handles the message (queue or send)

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar Ruud68
Ruud68 - comment - 21 Dec 2017

Just adding some thoughts: my hosting provider has a limit on how many emails I can sent per hour.
Currently I run the risk of mails not being sent because that limit was reached (I learned the hard way via mailing all users via the mass mail function :O)
It would be cool if the mail queue could be configured to these provider's thresholds


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19073.

avatar tonypartridge
tonypartridge - comment - 21 Dec 2017

Yep that’s one of the whole points of having a queuing system. Default limit should be 250 per hourly.

On 21 Dec 2017, 08:24 +0000, Ruud notifications@github.com, wrote:

Just adding some thoughts: my hosting provider has a limit on how many emails I can sent per hour.
Currently I run the risk of mails not being sent because that limit was reached (I learned the hard way via mailing all users via the mass mail function :O)
It would be cool if the mail queue could be configured to these provider's thresholds
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19073.

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar stutteringp0et
stutteringp0et - comment - 22 Dec 2017

I've got a CLI that implements limiting while verifying weblinks - it loads a configured maximum so as to not bog down the server. It's an integer field in a plugin config - super easy to do.

So, it stands to reason there should be a management component to hold configurations like this. I think it would be prudent to come up with a list of things the management app should do/track as that will help to define what the database should look like.

Off the top of my head, each mail record should include:

  1. user id of the user sending the mail (0 for automated emails)
    extension sending the email (com_users, com_contact, plg_system_whatever, etc)
  2. message priority (some core components are priority zero - most important, others can be optionally configured as 1 and higher - least important)
  3. DateTime the message was queued
  4. DateTime of the last send attempt (defaults to zeros)
  5. DateTime the message was successfully sent
  6. Number of attempts (JSON array of DateTime?)
  7. header and body (it's what PHPMailer can natively output and accept as input to send later)
  8. message id (checksum of the header maybe?)

Configurations for the CLI:
max messages per time period - max and time period both configurable
max messages per CLI run
max retries per message
retry interval
failed message recipient(s)

The success DateTime is used to determine the number of messages sent for the configured max/time period. Once the messages are no longer needed, they would be discarded (or not? why?)

avatar tonypartridge
tonypartridge - comment - 22 Dec 2017

Sounds good!

I would add a sent column and you can index on this too. This should then be used to get unsent messages.

Messages should stay in the system so the user has a log. I Would limit it to say 2,500 stores and on running of the send routine you can delete messages which are above the 2,500 limit which is overridable in the components Params

That makes 10 things 😉 you missed it no.2 in your list and made it part of no.1

On 22 Dec 2017, 06:41 +0000, Michael Richey notifications@github.com, wrote:

I've got a CLI that implements limiting while verifying weblinks - it loads a configured maximum so as to not bog down the server. It's an integer field in a plugin config - super easy to do.
So, it stands to reason there should be a management component to hold configurations like this. I think it would be prudent to come up with a list of things the management app should do/track as that will help to define what the database should look like.
Off the top of my head, each mail record should include:

  1. user id of the user sending the mail (0 for automated emails)
    extension sending the email (com_users, com_contact, plg_system_whatever, etc)
  2. message priority (some core components are priority zero - most important, others can be optionally configured as 1 and higher - least important)
  3. DateTime the message was queued
  4. DateTime of the last send attempt (defaults to zeros)
  5. DateTime the message was successfully sent
  6. Number of attempts (JSON array of DateTime?)
  7. header and body (it's what PHPMailer can natively output and accept as input to send later)
  8. message id (checksum of the header)

Configurations for the CLI:
max messages per time period - max and time period both configurable
max messages per CLI run
max retries per message
retry interval
failed message recipient(s)
The success DateTime is used to determine the number of messages sent for the configured max/time period. Once the messages are no longer needed, they would be discarded (or not? why?)

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar stutteringp0et
stutteringp0et - comment - 22 Dec 2017

Some messages need to be purged - like new user emails, because they contain a password in the clear. Password reset are one-time-use messages. Forgot username might become invalid if the user changes their name.... maybe a classification for sensitive messages that need to be purged.....possibly based on the sender extension? Like, anything coming from com_user should be purged upon send, to protect potentially sensitive information.

Maybe that's another configuration option, selecting which extensions emails are archived, while the rest are purged.

avatar franz-wohlkoenig franz-wohlkoenig - change - 29 Dec 2017
Build staging 4.0-dev
avatar shoulders
shoulders - comment - 13 Jan 2018

Just incase this has not been covered I would have a Global Joomla email cron/component where all 3rd parties can send their emails to the global que only if they choose that because some emails need to be sent out immediately.

This que could then control the send rate of all emails which and also give statistics on email volume and sources which is very much needed when running on shared hosting (i.e. most of us amateurs).

Developers would then not need to write their own crons and run the risk of us end users still overloading our servers email system. Consider you have easyblog with lots of emails being sent out and kunena with lots of emails being sent out by their seperate crons you have to work out and estimate totals from boths systems and make sure the combined lists dont go over your total or your accout will get suspended etc.., this can get tricky and requires constant revisits if you nare getting towards your max limit.

Most social components want to send emails nowadays.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19073.

avatar tonypartridge
tonypartridge - comment - 13 Jan 2018

Yep that’s exactly what was talked about basically, all that are sent straight away should still be logged :-)

On 13 Jan 2018, 11:57 +0000, shoulders notifications@github.com, wrote:

Just incase this has not been covered I would have a Global Joomla email cron/component where all 3rd parties can send their emails to the global que only if they choose that because some emails need to be sent out immediately.
This que could then control the send rate of all emails which and also give statistics on email volume and sources which is very much needed when running on shared hosting (i.e. most of us amateurs).
Developers would then not need to write their own crons and run the risk of us end users still overloading our servers email system. Consider you have easyblog with lots of emails being sent out and kunena with lots of emails being sent out by their seperate crons you have to work out and estimate totals from boths systems and make sure the combined lists dont go over your total or your accout will get suspended etc.., this can get tricky and requires constant revisits if you nare getting towards your max limit.
Most social components want to send emails nowadays.
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19073.

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar Ruud68
Ruud68 - comment - 13 Jan 2018

Any thoughts on if this is a change in Joomla Mailer class, or something extension developers need to implement?
I use e.g. Acymailing to sent newsletters (okay to be queued), but also to sent out the Joomla notification (new user, reset password etc.) these should not be queued / sent out immediately.
When the Mailer class 'catches' and queues all mail, then there should also be a way to tell that class what to do with what emails.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19073.
avatar roland-d
roland-d - comment - 13 Jan 2018

@stutteringp0et It would be a great feature to add to Joomla. I have implemented a mail queue in my own component that deletes obsolete users. From my experience this works fine using settings to fine-tune to server specs.

avatar tonypartridge
tonypartridge - comment - 13 Jan 2018

Options will need to be added in, and things like store text set to no for password emails etc.

All needs building in

On 13 Jan 2018, 14:18 +0000, RolandD notifications@github.com, wrote:

@stutteringp0et It would be a great feature to add to Joomla. I have implemented a mail queue in my own component that deletes obsolete users. From my experience this works fine using settings to fine-tune to server specs.

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar stutteringp0et
stutteringp0et - comment - 13 Jan 2018

The idea that I had regarding queuing was that the site administrator could turn it on/off globally. There would be a priority classification for emails like registration, password resets and that sort to be sent (virtually) immediately - at least, first in the next run (which would probably be once per-minute, as this is the smallest increment that CRON can handle)

Storing passwords in queue would be necessary until the email is sent - and then those messages would be deleted (or have their text replaced with a SENSITIVE DATA REMOVED message). I think that will be handled by the priority. Maybe something like 0 for sensitive, 1 for system, then everything else would be a priority 2. Sensitive emails get processed out of queue first, so the sensitive data doesn't remain in the database any longer than necessary. System goes next (lost password/username, system messages and stuff), then everything else.

I just got past a major deployment, and this is next on my list of things to do. I'll build a prototype which addresses these things and let you guys hammer on it.

Just so you understand where I'm at on this - I've got an extension which sends a half dozen emails on certain visitor generated events. It sucks that the user sits there for 5-10 seconds while the system sends emails. I thought about building a message queue into my extension, then I thought it might be more beneficial to the community if Joomla had its own queuing system. I figure, if I need a queuing system once, I'll probably need it again - and I'd rather build it only once.

avatar stutteringp0et
stutteringp0et - comment - 13 Jan 2018

@Ruud68 it would be a very minor change to the jmailer class. The modification would be to detect if the queuing system was enabled, and if so queue the message - otherwise it would send immediately. The modification would be found in the send() method (probably), because at that point - the message has been built and could be directed into the database easily.

avatar tonypartridge
tonypartridge - comment - 13 Jan 2018

Password and user emails need to always go immediately! And data never stored by default.

On 13 Jan 2018, 22:57 +0000, Michael Richey notifications@github.com, wrote:

@Ruud68 it would be a very minor change to the jmailer class. The modification would be to detect if the queuing system was enabled, and if so queue the message - otherwise it would send immediately. The modification would be found in the send() method (probably), because at that point - the message has been built and could be directed into the database easily.

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar mbabker
mbabker - comment - 13 Jan 2018

You can't have it both ways (push email data into a queue and not store that data in some way that can be read). Either some email types are just sent without touching the queue at all (and you don't have to worry about data management) or it goes into the queue and everything going into the queue is handled the same way.

avatar tonypartridge
tonypartridge - comment - 13 Jan 2018

You can, push it into the queue and mark it as sent. Emails for un/people have to be instant sends without storing personal information

On 13 Jan 2018, 23:19 +0000, Michael Babker notifications@github.com, wrote:

You can't have it both ways (push email data into a queue and not store that data in some way that can be read). Either some email types are just sent without touching the queue at all (and you don't have to worry about data management) or it goes into the queue and everything going into the queue is handled the same way.

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar mbabker
mbabker - comment - 13 Jan 2018

Putting something in a queue means it is scheduled to be done at a later time. If you're putting already complete jobs into the queue, then it defeats the point of a queue.

I think we're getting some wires crossed here. A job queue system doesn't really need to retain data about successful jobs (you can use other tools like logging to keep records of that), the queue system just needs to know what jobs to run and at what priority with what data at its simplest definition. Especially consider a highly active system where you wouldn't want a database table to be bloated with thousands of extra records that at that point serve as nothing but a logging system. This means a mail's content is immediately removed from storage as soon as the job for that message succeeds (if you're storing that mail content in a log, it's the API that's handling logging that's responsible for sanitizing it).

This is why for example Laravel's mailing API has two sending mechanisms, one to queue a message (which pushes the sending task into its native job queuing system) and one that immediately sends the message without touching the queue.

avatar stutteringp0et
stutteringp0et - comment - 13 Jan 2018

Someone had previously suggested an option to keep sent messages, a sort of running communication record. That's why there was going to be a method to determine sensitive messages which should not be stored.

avatar mbabker
mbabker - comment - 13 Jan 2018

Make it a supplemental logging system. What we are talking about here right now has IMO three distinct pieces:

  • A general job queue management system
  • A specific implementation surrounding email traffic
  • Transaction logging for email traffic
avatar stutteringp0et
stutteringp0et - comment - 14 Jan 2018

For a general job queue, would it be better to go with something already established and implement a worker interface in a Joomla CLI? I was only interested in a mail queue, but if a general job queue has a better chance of core adoption - I'm game.... I just wonder if it's necessary to re-invent the wheel when there are existing (and mature) job queue systems that could be interfaced.

avatar Ruud68
Ruud68 - comment - 14 Jan 2018

Either some email types are just sent without touching the queue at all (and you don't have to worry about data management) or it goes into the queue and everything going into the queue is handled the same way.

My hoster has an email limit per hour (siteground). Emails sent directly are counted by the hoster as are emails from the queue. So if I limit the queue to eg 400 per hour and mails are also send directly (not via queue) then these mails should also be counted for the mail queue limit.

That is why I think all emails should be 'routed' via the queue where direct emails are sent out immediately (without cron), but are counted by the queue in the configured limits.

As mails send from a domain per hour is becoming a relevant spam indicator, I think it is only a matter of time before all hosting companies apply a limit of some sort: shared hosting or not.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19073.

avatar tonypartridge
tonypartridge - comment - 14 Jan 2018

As above, it makes sense that the mail queue is a full mail routing log which queues mail when told too. A simple config toggle to force a queue all could be added, but to be perfectly honest if you all user registrations are your queueing their mail it would kill your site since it could be an hour till you get the mail, or even more if there are hundreds of queued mail.

It doesn’t need to be complicated;

  1. Mail queue filtered by not sent by default
  2. Limit total stored sent mail to say 5,000, configurable in the components config.
  3. Add in options to send mail; sent and store body.
  4. Hourly limit to send, default 200.

Allowing running the queue via plugin and cli.

This is how I have always envisioned doing it.

On 14 Jan 2018, 10:39 +0000, Ruud notifications@github.com, wrote:

Either some email types are just sent without touching the queue at all (and you don't have to worry about data management) or it goes into the queue and everything going into the queue is handled the same way.
My hoster has an email limit per hour (siteground). Emails sent directly are counted by the hoster as are emails from the queue. So if I limit the queue to eg 400 per hour and mails are also send directly (not via queue) then these mails should also be counted for the mail queue limit.
That is why I think all emails should be 'routed' via the queue where direct emails are sent out immediately (without cron), but are counted by the queue in the configured limits.
As mails send from a domain per hour is becoming a relevant spam indicator, I think it is only a matter of time before all hosting companies apply a limit of some sort: shared hosting or not.
This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19073.

You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

avatar mbabker
mbabker - comment - 14 Jan 2018

Again, you are talking specific implementation details for one specific problem.

The first thing Joomla needs is a job queuing system, in general. Not a solution specific for one task. Otherwise it really defeats the purpose of introducing such a system.

A job queuing system is not a transaction log. So speaking as a general job queuing system, no, it should not have a full transaction log where every action is recorded, to include things that are not even dispatched through the queue system.

So now you get into the mail specific details. This is where you get into the issue of having per-hour sending limits, transaction logging, and whatnot.

The mail system needs to be aware of the per-hour sending limit and keep track of this regardless of if a mail is sent as part of the job queuing system or directly sent without being queued. This specific feature is NOT a part of a job queue system and should not be integrated into one.

You are also looking for a transaction log which can store sent messages (and optionally sanitize their contents). This transaction log is a separate feature from the mail sending limitations AND a job queue system and should be handled separately.

Yes, all three of these things might need to be able to work with each other, but step back and look at it from a high level perspective and you will see that you have three distinctive (major) feature requests in this thread and each of them needs a proper implementation.

avatar mbabker
mbabker - comment - 14 Jan 2018

For a general job queue, would it be better to go with something already established and implement a worker interface in a Joomla CLI?

Yes. Let's not re-invent the wheel (I just finished this exercise for a project because the solution I needed only existed as a flawed extension and I had to re-implement it without all the flaws).

I was only interested in a mail queue, but if a general job queue has a better chance of core adoption - I'm game

I'm looking at things at a higher level. A job queue has use beyond just pushing email traffic out. If we're going to have a system that's enqueuing jobs to be run through another worker system (i.e. cron) then the high level stuff might as well be generic and usable for other tasks even if core itself isn't doing anything but queuing emails.

avatar stutteringp0et
stutteringp0et - comment - 14 Jan 2018

I think I can design something to fit this. I read about someone using redis as a job queue, and since J has a redis session store now, it seems like a likely candidate.

I'll do that research and come back with a plan for discussion. I'll frame the plan around the email implementation, but will make sure it's generic enough to handle any kind of queued task.

avatar mbabker
mbabker - comment - 14 Jan 2018

Depending how in depth we want to go, a few packages which are job queue systems in various contexts:

avatar stutteringp0et
stutteringp0et - comment - 14 Jan 2018

@tonypartridge - There isn't any reason for a registration email to take an hour to send out if routed through a job queue unless the queue worker can't keep up with the volume - in which case, you could launch a worker on another system. The system I envisioned for processing email would run via CRON every minute, delaying email up to one minute. If that isn't fast enough, I'm sure a clever person could write a daemon shell script that loops, listening for a queue event signal and runs jobs immediately.

@mbabker - Resque is the redis implementation I was reading about last night.

avatar mbabker
mbabker - comment - 14 Jan 2018

Resque might be OK, but just keep in mind we probably shouldn't lock the implementation to those having access to Redis (if you look at Laravel's system it has an abstraction layer for different connections such as Redis, a local database, and some other common system level integrations).

avatar brianteeman
brianteeman - comment - 14 Jan 2018

via CRON every minute

Hosts with restrictive mail limits are likely to have restrictive cron limits as well

avatar stutteringp0et
stutteringp0et - comment - 14 Jan 2018

@brianteeman - The plan was to make it optional.

avatar stutteringp0et
stutteringp0et - comment - 14 Jan 2018

@mbabker - illuminate it is - if queue system options are desired.

Add a Comment

Login with GitHub to post a comment