MailQueue module

[datakick]

Hi everyone,

I've releaesd new module MailQueue. This module uses latest thirty bees Mail Transport functionality, so currently it can be used in bleeding edge / 1.5.0 only.

Link to the module: https://store.getdatakick.com/en/modules/mail-queue

As the module name suggest, the main functionality of this module is to defer sending emails. Instead of trying to deliver email as part of business process, module will serialize and save it to storage (filesystem, redis, db). Cron or CLI queue processor (that runs in separate thread) will pick these queued emails and attempt delivery later.

You can create multiple outgoing queues, and create conditions to determine which emails should go to which queue. For example, queue for 'contact' form messages. Queue for shop admin recipients. Queue for emails that contains words 'binance' or 'viagra'. Queue for transaction emails, queue for marketing emails,...

Each queue can have different settings, of course (it wouldn't make it much sense otherwise)

• throttling -- how many emails can be sent per hour
• delay -- initial delay to send email 
• failed delivery retry settings 
• manual approval, etc 
• retention policy -- how long do you want to keep email in the queue / after send
• sending channel -- every queue can be send using different email transport. You can send transaction and marketing emails through differnt SMTP servers

Some benefits of this system

1. original process is not blocked by email sending. Slow SMTP servers can block order creation process even for a few seconds.
2. process is not interrupted by SMTP errors - if SMTP server was down, exception could be raised, and this could block process to be finalized (order not completed)
3. this supports sending a large amount of emails easily -- enqueueing emails is fast, so you can generate a lot of emails during one request - useful for sending marketing emails to a lot of recipients 
4. retry option - cron/cli can retry deliver email later if SMTP error occurs
5. you can view email content before it is sent, and even after it was sent (depending on queue retention policy)
6. delaying some emails (for example you can delay review request email by 3 days)
7. pause queue in case of some attack
8. email priority
9. different sending channels

I'm currently looking for volunteers that would like to test this module, before I release it officially. 

And of course, if you have some ideas or questions, let me know. I'd love to hear them all.

[the.rampage.rado]

Will this speed up the last step in the order?

[datakick]

26 minutes ago, the.rampage.rado said:

Will this speed up the last step in the order?

yes

[the.rampage.rado]

Nice, I would like to test it and then buy it just to fix that. 🙂

[datakick]

I've just officially released this module, you can find it here: https://store.getdatakick.com/en/modules/mail-queue

This module requires thirty bees 1.5.0 or newer

This thirty bees module can help you manage your outgoing emails easily. Instead of sending emails immediately, module will store them inside one of the mail queues. You can create rules to determine which queue will be selected for every email.

Module cron job will later attempt to deliver emails from queues. This alone is a great performance improvement. Sending emails can take a lot of time. Because it is synchronous, it blocks response. By offloading this task from customer initiated request to backend thread, you will make your front office much faster. 

But there are other benefits of using mail queues:

Failure recovery

When attempt to send send email fails for any reasons, module will retry it later. Without mail queue moule, such failure would raise error/exception and would broke functionality. For example, email sending failure could block displaying order confirmation page, or processing payment. This will never happen again

Email preview

You can click on every email in your queues and see the content, download attachments, etc. You can easily see what will be, or what was actually sent to your customers. 

Sending delay

You can specify initial delay -- minimum amount of time the email will sit inside the mail queue before module will attempt to deliver it.

This can be useful, for example, to decouple emails - Order Confirmation and Bank wire payment emails are usually send together. You can create dedicated queue for bank wire payment email and delay these emails by few minutes. 

Manual moderation

When you pause email queue, module cron will not send emails from that queue automatically. You can manually review all pending emails, and delete or send them.

This can be useful tool to fight spam. You can create paused queue, and define conditions like "put email into this queue if it contains text 'online casino' or 'viagra'"   

Audit log

You can set retention policy per queue. Sent emails can be immediately deleted from queue, or kept forever or for specified amount of days.  

Mass emailing

Some modules can send many emails at once (for example email marketing modules). If implemented improperly, such task can timeout. With mail queues it will not, because saving emails locally is much faster than actually sending them. You can put hundreds or even a thousands of emails into the queue in one requests.

Throttling 

You can set sending rate per queue, for example to send max 3 emails per hour. This can be useful -- sending many emails at once can put your domain on blacklist. By throttling outgoing emails you can prevent this

Different email transports

You can set different email transport for every queue.

With this, you can use dedicated SMTP server for your transaction emails, and different one for you marketing emails. When your marketing emails are marked as spam, it will not have any impact on your transaction emails reputations.

[30knees]

20 hours ago, datakick said:

This alone is a great performance improvement. Sending emails can take a lot of time. Because it is synchronous, it blocks response. By offloading this task from customer initiated request to backend thread, you will make your front office much faster. 

I am curious about this. Does that mean that eg all order confirmation emails should typically have a minimal delay? 

[datakick]

1 hour ago, 30knees said:

I am curious about this. Does that mean that eg all order confirmation emails should typically have a minimal delay? 

yes, the delay depends on cron frequency. If you run it every minute, the longest delay should be a minute.

You can also set additional delay per queue, if you need some emails to be send later. For example, to decouple order confirmation email and bank wire email.

[30knees]

I have a delay of 0, which means that the delay will depend on the cron job. Is this correct? Because at the moment the default queue isn't delaying anything, as far as I can tell.

On 4/6/2024 at 9:09 AM, datakick said:

yes, the delay depends on cron frequency. If you run it every minute, the longest delay should be a minute.

 

[datakick]

18 minutes ago, 30knees said:

I have a delay of 0, which means that the delay will depend on the cron job. Is this correct? Because at the moment the default queue isn't delaying anything, as far as I can tell.

 

You have to set mailqueue as your default email transport in Email settings, otherwise the system will send it immediately. 

[30knees]

One feature suggestion would be to be able to bulk select mails in the queue for processing, that is either delete or send now. 

Otherwise: All works well!

Edited May 28 by 30knees